1 Commits

Author SHA1 Message Date
8b2268fcc9 fix: improve async validator handling in Pubsub class
Signed-off-by: varun-r-mallya <varunrmallya@gmail.com>
2025-06-26 19:56:53 +05:30
88 changed files with 474 additions and 6048 deletions

View File

@ -1,64 +0,0 @@
mDNS Peer Discovery Example
===========================
This example demonstrates how to use mDNS (Multicast DNS) for peer discovery in py-libp2p.
Prerequisites
-------------
First, ensure you have py-libp2p installed and your environment is activated:
.. code-block:: console
$ python -m pip install libp2p
Running the Example
-------------------
The mDNS demo script allows you to discover peers on your local network using mDNS. To start a peer, run:
.. code-block:: console
$ mdns-demo
You should see output similar to:
.. code-block:: console
Run this from another console to start another peer on a different port:
python mdns-demo -p <ANOTHER_PORT>
Waiting for mDNS peer discovery events...
2025-06-20 23:28:12,052 - libp2p.example.discovery.mdns - INFO - Starting peer Discovery
To discover peers, open another terminal and run the same command with a different port:
.. code-block:: console
$ python mdns-demo -p 9001
You should see output indicating that a new peer has been discovered:
.. code-block:: console
Run this from the same folder in another console to start another peer on a different port:
python mdns-demo -p <ANOTHER_PORT>
Waiting for mDNS peer discovery events...
2025-06-20 23:43:43,786 - libp2p.example.discovery.mdns - INFO - Starting peer Discovery
2025-06-20 23:43:43,790 - libp2p.example.discovery.mdns - INFO - Discovered: 16Uiu2HAmGxy5NdQEjZWtrYUMrzdp3Syvg7MB2E5Lx8weA9DanYxj
When a new peer is discovered, its peer ID will be printed in the console output.
How it Works
------------
- Each node advertises itself on the local network using mDNS.
- When a new peer is discovered, the handler prints its peer ID.
- This is useful for local peer discovery without requiring a DHT or bootstrap nodes.
You can modify the script to perform additional actions when peers are discovered, such as opening streams or exchanging messages.

View File

@ -13,4 +13,3 @@ Examples
examples.pubsub
examples.circuit_relay
examples.kademlia
examples.mDNS

View File

@ -1,21 +0,0 @@
libp2p.discovery.events package
===============================
Submodules
----------
libp2p.discovery.events.peerDiscovery module
--------------------------------------------
.. automodule:: libp2p.discovery.events.peerDiscovery
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: libp2p.discovery.events
:members:
:undoc-members:
:show-inheritance:

View File

@ -1,45 +0,0 @@
libp2p.discovery.mdns package
=============================
Submodules
----------
libp2p.discovery.mdns.broadcaster module
----------------------------------------
.. automodule:: libp2p.discovery.mdns.broadcaster
:members:
:undoc-members:
:show-inheritance:
libp2p.discovery.mdns.listener module
-------------------------------------
.. automodule:: libp2p.discovery.mdns.listener
:members:
:undoc-members:
:show-inheritance:
libp2p.discovery.mdns.mdns module
---------------------------------
.. automodule:: libp2p.discovery.mdns.mdns
:members:
:undoc-members:
:show-inheritance:
libp2p.discovery.mdns.utils module
----------------------------------
.. automodule:: libp2p.discovery.mdns.utils
:members:
:undoc-members:
:show-inheritance:
Module contents
---------------
.. automodule:: libp2p.discovery.mdns
:members:
:undoc-members:
:show-inheritance:

View File

@ -1,22 +0,0 @@
libp2p.discovery package
========================
Subpackages
-----------
.. toctree::
:maxdepth: 4
libp2p.discovery.events
libp2p.discovery.mdns
Submodules
----------
Module contents
---------------
.. automodule:: libp2p.discovery
:members:
:undoc-members:
:show-inheritance:

View File

@ -8,7 +8,6 @@ Subpackages
:maxdepth: 4
libp2p.crypto
libp2p.discovery
libp2p.host
libp2p.identity
libp2p.io

View File

@ -3,65 +3,6 @@ Release Notes
.. towncrier release notes start
py-libp2p v0.2.9 (2025-07-09)
-----------------------------
Breaking Changes
~~~~~~~~~~~~~~~~
- Reordered the arguments to ``upgrade_security`` to place ``is_initiator`` before ``peer_id``, and made ``peer_id`` optional.
This allows the method to reflect the fact that peer identity is not required for inbound connections. (`#681 <https://github.com/libp2p/py-libp2p/issues/681>`__)
Bugfixes
~~~~~~~~
- Add timeout wrappers in:
1. ``multiselect.py``: ``negotiate`` function
2. ``multiselect_client.py``: ``select_one_of`` , ``query_multistream_command`` functions
to prevent indefinite hangs when a remote peer does not respond. (`#696 <https://github.com/libp2p/py-libp2p/issues/696>`__)
- Align stream creation logic with yamux specification (`#701 <https://github.com/libp2p/py-libp2p/issues/701>`__)
- Fixed an issue in ``Pubsub`` where async validators were not handled reliably under concurrency. Now uses a safe aggregator list for consistent behavior. (`#702 <https://github.com/libp2p/py-libp2p/issues/702>`__)
Features
~~~~~~~~
- Added support for ``Kademlia DHT`` in py-libp2p. (`#579 <https://github.com/libp2p/py-libp2p/issues/579>`__)
- Limit concurrency in ``push_identify_to_peers`` to prevent resource congestion under high peer counts. (`#621 <https://github.com/libp2p/py-libp2p/issues/621>`__)
- Store public key and peer ID in peerstore during handshake
Modified the InsecureTransport class to accept an optional peerstore parameter and updated the handshake process to store the received public key and peer ID in the peerstore when available.
Added test cases to verify:
1. The peerstore remains unchanged when handshake fails due to peer ID mismatch
2. The handshake correctly adds a public key to a peer ID that already exists in the peerstore but doesn't have a public key yet (`#631 <https://github.com/libp2p/py-libp2p/issues/631>`__)
- Fixed several flow-control and concurrency issues in the ``YamuxStream`` class. Previously, stress-testing revealed that transferring data over ``DEFAULT_WINDOW_SIZE`` would break the stream due to inconsistent window update handling and lock management. The fixes include:
- Removed sending of window updates during writes to maintain correct flow-control.
- Added proper timeout handling when releasing and acquiring locks to prevent concurrency errors.
- Corrected the ``read`` function to properly handle window updates for both ``read_until_EOF`` and ``read_n_bytes``.
- Added event logging at ``send_window_updates`` and ``waiting_for_window_updates`` for better observability. (`#639 <https://github.com/libp2p/py-libp2p/issues/639>`__)
- Added support for ``Multicast DNS`` in py-libp2p (`#649 <https://github.com/libp2p/py-libp2p/issues/649>`__)
- Optimized pubsub publishing to send multiple topics in a single message instead of separate messages per topic. (`#685 <https://github.com/libp2p/py-libp2p/issues/685>`__)
- Optimized pubsub message writing by implementing a write_msg() method that uses pre-allocated buffers and single write operations, improving performance by eliminating separate varint prefix encoding and write operations in FloodSub and GossipSub. (`#687 <https://github.com/libp2p/py-libp2p/issues/687>`__)
- Added peer exchange and backoff logic as part of Gossipsub v1.1 upgrade (`#690 <https://github.com/libp2p/py-libp2p/issues/690>`__)
Internal Changes - for py-libp2p Contributors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Added sparse connect utility function to pubsub test utilities for creating test networks with configurable connectivity. (`#679 <https://github.com/libp2p/py-libp2p/issues/679>`__)
- Added comprehensive tests for pubsub connection utility functions to verify degree limits are enforced, excess peers are handled correctly, and edge cases (degree=0, negative values, empty lists) are managed gracefully. (`#707 <https://github.com/libp2p/py-libp2p/issues/707>`__)
- Added extra tests for identify push concurrency cap under high peer load (`#708 <https://github.com/libp2p/py-libp2p/issues/708>`__)
Miscellaneous Changes
~~~~~~~~~~~~~~~~~~~~~
- `#678 <https://github.com/libp2p/py-libp2p/issues/678>`__, `#684 <https://github.com/libp2p/py-libp2p/issues/684>`__
py-libp2p v0.2.8 (2025-06-10)
-----------------------------

View File

@ -8,10 +8,9 @@ import trio
from libp2p import (
new_host,
)
from libp2p.identity.identify.identify import (
ID as IDENTIFY_PROTOCOL_ID,
identify_handler_for,
parse_identify_response,
from libp2p.identity.identify.identify import ID as IDENTIFY_PROTOCOL_ID
from libp2p.identity.identify.pb.identify_pb2 import (
Identify,
)
from libp2p.peer.peerinfo import (
info_from_p2p_addr,
@ -51,7 +50,7 @@ def print_identify_response(identify_response):
)
async def run(port: int, destination: str, use_varint_format: bool = True) -> None:
async def run(port: int, destination: str) -> None:
localhost_ip = "0.0.0.0"
if not destination:
@ -59,24 +58,11 @@ async def run(port: int, destination: str, use_varint_format: bool = True) -> No
listen_addr = multiaddr.Multiaddr(f"/ip4/{localhost_ip}/tcp/{port}")
host_a = new_host()
# Set up identify handler with specified format
identify_handler = identify_handler_for(
host_a, use_varint_format=use_varint_format
)
host_a.set_stream_handler(IDENTIFY_PROTOCOL_ID, identify_handler)
async with host_a.run(listen_addrs=[listen_addr]):
# Get the actual address and replace 0.0.0.0 with 127.0.0.1 for client
# connections
server_addr = str(host_a.get_addrs()[0])
client_addr = server_addr.replace("/ip4/0.0.0.0/", "/ip4/127.0.0.1/")
format_name = "length-prefixed" if use_varint_format else "raw protobuf"
print(
f"First host listening (using {format_name} format). "
f"Run this from another console:\n\n"
"First host listening. Run this from another console:\n\n"
f"identify-demo "
f"-d {client_addr}\n"
f"-d {host_a.get_addrs()[0]}\n"
)
print("Waiting for incoming identify request...")
await trio.sleep_forever()
@ -98,18 +84,11 @@ async def run(port: int, destination: str, use_varint_format: bool = True) -> No
try:
print("Starting identify protocol...")
# Read the complete response (could be either format)
# Read a larger chunk to get all the data before stream closes
response = await stream.read(8192) # Read enough data in one go
response = await stream.read()
await stream.close()
# Parse the response using the robust protocol-level function
# This handles both old and new formats automatically
identify_msg = parse_identify_response(response)
identify_msg = Identify()
identify_msg.ParseFromString(response)
print_identify_response(identify_msg)
except Exception as e:
print(f"Identify protocol error: {e}")
@ -119,12 +98,9 @@ async def run(port: int, destination: str, use_varint_format: bool = True) -> No
def main() -> None:
description = """
This program demonstrates the libp2p identify protocol.
First run 'identify-demo -p <PORT> [--raw-format]' to start a listener.
First run identify-demo -p <PORT>' to start a listener.
Then run 'identify-demo <ANOTHER_PORT> -d <DESTINATION>'
where <DESTINATION> is the multiaddress shown by the listener.
Use --raw-format to send raw protobuf messages (old format) instead of
length-prefixed protobuf messages (new format, default).
"""
example_maddr = (
@ -139,22 +115,10 @@ def main() -> None:
type=str,
help=f"destination multiaddr string, e.g. {example_maddr}",
)
parser.add_argument(
"--raw-format",
action="store_true",
help=(
"use raw protobuf format (old format) instead of "
"length-prefixed (new format)"
),
)
args = parser.parse_args()
# Determine format: raw format if --raw-format is specified, otherwise
# length-prefixed
use_varint_format = not args.raw_format
try:
trio.run(run, *(args.port, args.destination, use_varint_format))
trio.run(run, *(args.port, args.destination))
except KeyboardInterrupt:
pass

View File

@ -57,56 +57,18 @@ from libp2p.peer.peerinfo import (
logger = logging.getLogger("libp2p.identity.identify-push-example")
def custom_identify_push_handler_for(host, use_varint_format: bool = True):
def custom_identify_push_handler_for(host):
"""
Create a custom handler for the identify/push protocol that logs and prints
the identity information received from the dialer.
Args:
host: The libp2p host
use_varint_format: If True, expect length-prefixed format; if False, expect
raw protobuf
"""
async def handle_identify_push(stream: INetStream) -> None:
peer_id = stream.muxed_conn.peer_id
try:
if use_varint_format:
# Read length-prefixed identify message from the stream
from libp2p.utils.varint import decode_varint_from_bytes
# First read the varint length prefix
length_bytes = b""
while True:
b = await stream.read(1)
if not b:
break
length_bytes += b
if b[0] & 0x80 == 0:
break
if not length_bytes:
logger.warning("No length prefix received from peer %s", peer_id)
return
msg_length = decode_varint_from_bytes(length_bytes)
# Read the protobuf message
data = await stream.read(msg_length)
if len(data) != msg_length:
logger.warning("Incomplete message received from peer %s", peer_id)
return
else:
# Read raw protobuf message from the stream
data = b""
while True:
chunk = await stream.read(4096)
if not chunk:
break
data += chunk
# Read the identify message from the stream
data = await stream.read()
identify_msg = Identify()
identify_msg.ParseFromString(data)
@ -167,13 +129,9 @@ def custom_identify_push_handler_for(host, use_varint_format: bool = True):
return handle_identify_push
async def run_listener(port: int, use_varint_format: bool = True) -> None:
async def run_listener(port: int) -> None:
"""Run a host in listener mode."""
format_name = "length-prefixed" if use_varint_format else "raw protobuf"
print(
f"\n==== Starting Identify-Push Listener on port {port} "
f"(using {format_name} format) ====\n"
)
print(f"\n==== Starting Identify-Push Listener on port {port} ====\n")
# Create key pair for the listener
key_pair = create_new_key_pair()
@ -181,14 +139,9 @@ async def run_listener(port: int, use_varint_format: bool = True) -> None:
# Create the listener host
host = new_host(key_pair=key_pair)
# Set up the identify and identify/push handlers with specified format
host.set_stream_handler(
ID_IDENTIFY, identify_handler_for(host, use_varint_format=use_varint_format)
)
host.set_stream_handler(
ID_IDENTIFY_PUSH,
identify_push_handler_for(host, use_varint_format=use_varint_format),
)
# Set up the identify and identify/push handlers
host.set_stream_handler(ID_IDENTIFY, identify_handler_for(host))
host.set_stream_handler(ID_IDENTIFY_PUSH, custom_identify_push_handler_for(host))
# Start listening
listen_addr = multiaddr.Multiaddr(f"/ip4/0.0.0.0/tcp/{port}")
@ -212,15 +165,9 @@ async def run_listener(port: int, use_varint_format: bool = True) -> None:
await trio.sleep_forever()
async def run_dialer(
port: int, destination: str, use_varint_format: bool = True
) -> None:
async def run_dialer(port: int, destination: str) -> None:
"""Run a host in dialer mode that connects to a listener."""
format_name = "length-prefixed" if use_varint_format else "raw protobuf"
print(
f"\n==== Starting Identify-Push Dialer on port {port} "
f"(using {format_name} format) ====\n"
)
print(f"\n==== Starting Identify-Push Dialer on port {port} ====\n")
# Create key pair for the dialer
key_pair = create_new_key_pair()
@ -228,14 +175,9 @@ async def run_dialer(
# Create the dialer host
host = new_host(key_pair=key_pair)
# Set up the identify and identify/push handlers with specified format
host.set_stream_handler(
ID_IDENTIFY, identify_handler_for(host, use_varint_format=use_varint_format)
)
host.set_stream_handler(
ID_IDENTIFY_PUSH,
identify_push_handler_for(host, use_varint_format=use_varint_format),
)
# Set up the identify and identify/push handlers
host.set_stream_handler(ID_IDENTIFY, identify_handler_for(host))
host.set_stream_handler(ID_IDENTIFY_PUSH, identify_push_handler_for(host))
# Start listening on a different port
listen_addr = multiaddr.Multiaddr(f"/ip4/0.0.0.0/tcp/{port}")
@ -264,9 +206,7 @@ async def run_dialer(
try:
# Call push_identify_to_peer which returns a boolean
success = await push_identify_to_peer(
host, peer_info.peer_id, use_varint_format=use_varint_format
)
success = await push_identify_to_peer(host, peer_info.peer_id)
if success:
logger.info("Identify push completed successfully!")
@ -300,40 +240,29 @@ def main() -> None:
This program demonstrates the libp2p identify/push protocol.
Without arguments, it runs as a listener on random port.
With -d parameter, it runs as a dialer on random port.
Use --raw-format to send raw protobuf messages (old format) instead of
length-prefixed protobuf messages (new format, default).
"""
example = (
"/ip4/127.0.0.1/tcp/8000/p2p/QmQn4SwGkDZKkUEpBRBvTmheQycxAHJUNmVEnjA2v1qe8Q"
)
parser = argparse.ArgumentParser(description=description)
parser.add_argument("-p", "--port", default=0, type=int, help="source port number")
parser.add_argument(
"-d",
"--destination",
type=str,
help="destination multiaddr string",
)
parser.add_argument(
"--raw-format",
action="store_true",
help=(
"use raw protobuf format (old format) instead of "
"length-prefixed (new format)"
),
help=f"destination multiaddr string, e.g. {example}",
)
args = parser.parse_args()
# Determine format: raw format if --raw-format is specified, otherwise
# length-prefixed
use_varint_format = not args.raw_format
try:
if args.destination:
# Run in dialer mode with random available port if not specified
trio.run(run_dialer, args.port, args.destination, use_varint_format)
trio.run(run_dialer, args.port, args.destination)
else:
# Run in listener mode with random available port if not specified
trio.run(run_listener, args.port, use_varint_format)
trio.run(run_listener, args.port)
except KeyboardInterrupt:
print("\nInterrupted by user")
logger.info("Interrupted by user")

View File

@ -1,74 +0,0 @@
import argparse
import logging
import secrets
import multiaddr
import trio
from libp2p import (
new_host,
)
from libp2p.abc import PeerInfo
from libp2p.crypto.secp256k1 import (
create_new_key_pair,
)
from libp2p.discovery.events.peerDiscovery import peerDiscovery
logger = logging.getLogger("libp2p.discovery.mdns")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(
logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
)
logger.addHandler(handler)
# Set root logger to DEBUG to capture all logs from dependencies
logging.getLogger().setLevel(logging.DEBUG)
def onPeerDiscovery(peerinfo: PeerInfo):
logger.info(f"Discovered: {peerinfo.peer_id}")
async def run(port: int) -> None:
secret = secrets.token_bytes(32)
key_pair = create_new_key_pair(secret)
listen_addr = multiaddr.Multiaddr(f"/ip4/0.0.0.0/tcp/{port}")
peerDiscovery.register_peer_discovered_handler(onPeerDiscovery)
print(
"Run this from the same folder in another console to "
"start another peer on a different port:\n\n"
"mdns-demo -p <ANOTHER_PORT>\n"
)
print("Waiting for mDNS peer discovery events...\n")
logger.info("Starting peer Discovery")
host = new_host(key_pair=key_pair, enable_mDNS=True)
async with host.run(listen_addrs=[listen_addr]):
await trio.sleep_forever()
def main() -> None:
description = """
This program demonstrates mDNS peer discovery using libp2p.
To use it, run 'mdns-demo -p <PORT>', where <PORT> is the port number.
Start multiple peers on different ports to see discovery in action.
"""
parser = argparse.ArgumentParser(description=description)
parser.add_argument("-p", "--port", default=0, type=int, help="source port number")
parser.add_argument(
"-v", "--verbose", action="store_true", help="Enable verbose output"
)
args = parser.parse_args()
if args.verbose:
logger.setLevel(logging.DEBUG)
try:
trio.run(run, args.port)
except KeyboardInterrupt:
logger.info("Exiting...")
if __name__ == "__main__":
main()

View File

@ -32,9 +32,6 @@ from libp2p.custom_types import (
TProtocol,
TSecurityOptions,
)
from libp2p.discovery.mdns.mdns import (
MDNSDiscovery,
)
from libp2p.host.basic_host import (
BasicHost,
)
@ -84,8 +81,6 @@ DEFAULT_MUXER = "YAMUX"
# Multiplexer options
MUXER_YAMUX = "YAMUX"
MUXER_MPLEX = "MPLEX"
DEFAULT_NEGOTIATE_TIMEOUT = 5
def set_default_muxer(muxer_name: Literal["YAMUX", "MPLEX"]) -> None:
@ -250,8 +245,6 @@ def new_host(
disc_opt: IPeerRouting | None = None,
muxer_preference: Literal["YAMUX", "MPLEX"] | None = None,
listen_addrs: Sequence[multiaddr.Multiaddr] | None = None,
enable_mDNS: bool = False,
negotiate_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
) -> IHost:
"""
Create a new libp2p host based on the given parameters.
@ -263,7 +256,6 @@ def new_host(
:param disc_opt: optional discovery
:param muxer_preference: optional explicit muxer preference
:param listen_addrs: optional list of multiaddrs to listen on
:param enable_mDNS: whether to enable mDNS discovery
:return: return a host instance
"""
swarm = new_swarm(
@ -276,7 +268,8 @@ def new_host(
)
if disc_opt is not None:
return RoutedHost(swarm, disc_opt, enable_mDNS)
return BasicHost(network=swarm,enable_mDNS=enable_mDNS , negotitate_timeout=negotiate_timeout)
return RoutedHost(swarm, disc_opt)
return BasicHost(swarm)
__version__ = __version("libp2p")

View File

@ -50,11 +50,6 @@ if TYPE_CHECKING:
Pubsub,
)
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from libp2p.protocol_muxer.multiselect import Multiselect
from libp2p.pubsub.pb import (
rpc_pb2,
)
@ -390,18 +385,6 @@ class IPeerMetadata(ABC):
:raises Exception: If the operation is unsuccessful.
"""
@abstractmethod
def clear_metadata(self, peer_id: ID) -> None:
"""
Remove all stored metadata for the specified peer.
Parameters
----------
peer_id : ID
The peer identifier whose metadata are to be removed.
"""
# -------------------------- addrbook interface.py --------------------------
@ -493,272 +476,10 @@ class IAddrBook(ABC):
"""
# -------------------------- keybook interface.py --------------------------
class IKeyBook(ABC):
"""
Interface for an key book.
Provides methods for managing cryptographic keys.
"""
@abstractmethod
def pubkey(self, peer_id: ID) -> PublicKey:
"""
Returns the public key of the specified peer
Parameters
----------
peer_id : ID
The peer identifier whose public key is to be returned.
"""
@abstractmethod
def privkey(self, peer_id: ID) -> PrivateKey:
"""
Returns the private key of the specified peer
Parameters
----------
peer_id : ID
The peer identifier whose private key is to be returned.
"""
@abstractmethod
def add_pubkey(self, peer_id: ID, pubkey: PublicKey) -> None:
"""
Adds the public key for a specified peer
Parameters
----------
peer_id : ID
The peer identifier whose public key is to be added
pubkey: PublicKey
The public key of the peer
"""
@abstractmethod
def add_privkey(self, peer_id: ID, privkey: PrivateKey) -> None:
"""
Adds the private key for a specified peer
Parameters
----------
peer_id : ID
The peer identifier whose private key is to be added
privkey: PrivateKey
The private key of the peer
"""
@abstractmethod
def add_key_pair(self, peer_id: ID, key_pair: KeyPair) -> None:
"""
Adds the key pair for a specified peer
Parameters
----------
peer_id : ID
The peer identifier whose key pair is to be added
key_pair: KeyPair
The key pair of the peer
"""
@abstractmethod
def peer_with_keys(self) -> list[ID]:
"""Returns all the peer IDs stored in the AddrBook"""
@abstractmethod
def clear_keydata(self, peer_id: ID) -> None:
"""
Remove all stored keydata for the specified peer.
Parameters
----------
peer_id : ID
The peer identifier whose keys are to be removed.
"""
# -------------------------- metrics interface.py --------------------------
class IMetrics(ABC):
"""
Interface for metrics of peer interaction.
Provides methods for managing the metrics.
"""
@abstractmethod
def record_latency(self, peer_id: ID, RTT: float) -> None:
"""
Records a new round-trip time (RTT) latency value for the specified peer
using Exponentially Weighted Moving Average (EWMA).
Parameters
----------
peer_id : ID
The identifier of the peer for which latency is being recorded.
RTT : float
The round-trip time latency value to record.
"""
@abstractmethod
def latency_EWMA(self, peer_id: ID) -> float:
"""
Returns the current latency value for the specified peer using
Exponentially Weighted Moving Average (EWMA).
Parameters
----------
peer_id : ID
The identifier of the peer whose latency EWMA is to be returned.
"""
@abstractmethod
def clear_metrics(self, peer_id: ID) -> None:
"""
Clears the stored latency metrics for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose latency metrics are to be cleared.
"""
# -------------------------- protobook interface.py --------------------------
class IProtoBook(ABC):
"""
Interface for a protocol book.
Provides methods for managing the list of supported protocols.
"""
@abstractmethod
def get_protocols(self, peer_id: ID) -> list[str]:
"""
Returns the list of protocols associated with the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose supported protocols are to be returned.
"""
@abstractmethod
def add_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Adds the given protocols to the specified peer's protocol list.
Parameters
----------
peer_id : ID
The identifier of the peer to which protocols will be added.
protocols : Sequence[str]
A sequence of protocol strings to add.
"""
@abstractmethod
def set_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Replaces the existing protocols of the specified peer with the given list.
Parameters
----------
peer_id : ID
The identifier of the peer whose protocols are to be set.
protocols : Sequence[str]
A sequence of protocol strings to assign.
"""
@abstractmethod
def remove_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Removes the specified protocols from the peer's protocol list.
Parameters
----------
peer_id : ID
The identifier of the peer from which protocols will be removed.
protocols : Sequence[str]
A sequence of protocol strings to remove.
"""
@abstractmethod
def supports_protocols(self, peer_id: ID, protocols: Sequence[str]) -> list[str]:
"""
Returns the list of protocols from the input sequence that the peer supports.
Parameters
----------
peer_id : ID
The identifier of the peer to check for protocol support.
protocols : Sequence[str]
A sequence of protocol strings to check against the peer's
supported protocols.
"""
@abstractmethod
def first_supported_protocol(self, peer_id: ID, protocols: Sequence[str]) -> str:
"""
Returns the first protocol from the input list that the peer supports.
Parameters
----------
peer_id : ID
The identifier of the peer to check for supported protocols.
protocols : Sequence[str]
A sequence of protocol strings to check.
Returns
-------
str
The first matching protocol string, or an empty string
if none are supported.
"""
@abstractmethod
def clear_protocol_data(self, peer_id: ID) -> None:
"""
Clears all protocol data associated with the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose protocol data will be cleared.
"""
# -------------------------- peerstore interface.py --------------------------
class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
class IPeerStore(IAddrBook, IPeerMetadata):
"""
Interface for a peer store.
@ -766,7 +487,85 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
management, protocol handling, and key storage.
"""
# -------METADATA---------
@abstractmethod
def peer_info(self, peer_id: ID) -> PeerInfo:
"""
Retrieve the peer information for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
Returns
-------
PeerInfo
The peer information object for the given peer.
"""
@abstractmethod
def get_protocols(self, peer_id: ID) -> list[str]:
"""
Retrieve the protocols associated with the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
Returns
-------
list[str]
A list of protocol identifiers.
Raises
------
PeerStoreError
If the peer ID is not found.
"""
@abstractmethod
def add_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Add additional protocols for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
protocols : Sequence[str]
The protocols to add.
"""
@abstractmethod
def set_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Set the protocols for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
protocols : Sequence[str]
The protocols to set.
"""
@abstractmethod
def peer_ids(self) -> list[ID]:
"""
Retrieve all peer identifiers stored in the peer store.
Returns
-------
list[ID]
A list of all peer IDs in the store.
"""
@abstractmethod
def get(self, peer_id: ID, key: str) -> Any:
"""
@ -807,19 +606,6 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
"""
@abstractmethod
def clear_metadata(self, peer_id: ID) -> None:
"""
Clears the stored latency metrics for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose latency metrics are to be cleared.
"""
# --------ADDR-BOOK---------
@abstractmethod
def add_addr(self, peer_id: ID, addr: Multiaddr, ttl: int) -> None:
"""
@ -893,7 +679,25 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
"""
# --------KEY-BOOK----------
@abstractmethod
def add_pubkey(self, peer_id: ID, pubkey: PublicKey) -> None:
"""
Add a public key for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
pubkey : PublicKey
The public key to add.
Raises
------
PeerStoreError
If the peer already has a public key set.
"""
@abstractmethod
def pubkey(self, peer_id: ID) -> PublicKey:
"""
@ -916,6 +720,25 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
"""
@abstractmethod
def add_privkey(self, peer_id: ID, privkey: PrivateKey) -> None:
"""
Add a private key for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
privkey : PrivateKey
The private key to add.
Raises
------
PeerStoreError
If the peer already has a private key set.
"""
@abstractmethod
def privkey(self, peer_id: ID) -> PrivateKey:
"""
@ -938,44 +761,6 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
"""
@abstractmethod
def add_pubkey(self, peer_id: ID, pubkey: PublicKey) -> None:
"""
Add a public key for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
pubkey : PublicKey
The public key to add.
Raises
------
PeerStoreError
If the peer already has a public key set.
"""
@abstractmethod
def add_privkey(self, peer_id: ID, privkey: PrivateKey) -> None:
"""
Add a private key for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
privkey : PrivateKey
The private key to add.
Raises
------
PeerStoreError
If the peer already has a private key set.
"""
@abstractmethod
def add_key_pair(self, peer_id: ID, key_pair: KeyPair) -> None:
"""
@ -995,213 +780,6 @@ class IPeerStore(IPeerMetadata, IAddrBook, IKeyBook, IMetrics, IProtoBook):
"""
@abstractmethod
def peer_with_keys(self) -> list[ID]:
"""Returns all the peer IDs stored in the AddrBook"""
@abstractmethod
def clear_keydata(self, peer_id: ID) -> None:
"""
Remove all stored keydata for the specified peer.
Parameters
----------
peer_id : ID
The peer identifier whose keys are to be removed.
"""
# -------METRICS---------
@abstractmethod
def record_latency(self, peer_id: ID, RTT: float) -> None:
"""
Records a new round-trip time (RTT) latency value for the specified peer
using Exponentially Weighted Moving Average (EWMA).
Parameters
----------
peer_id : ID
The identifier of the peer for which latency is being recorded.
RTT : float
The round-trip time latency value to record.
"""
@abstractmethod
def latency_EWMA(self, peer_id: ID) -> float:
"""
Returns the current latency value for the specified peer using
Exponentially Weighted Moving Average (EWMA).
Parameters
----------
peer_id : ID
The identifier of the peer whose latency EWMA is to be returned.
"""
@abstractmethod
def clear_metrics(self, peer_id: ID) -> None:
"""
Clears the stored latency metrics for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose latency metrics are to be cleared.
"""
# --------PROTO-BOOK----------
@abstractmethod
def get_protocols(self, peer_id: ID) -> list[str]:
"""
Retrieve the protocols associated with the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
Returns
-------
list[str]
A list of protocol identifiers.
Raises
------
PeerStoreError
If the peer ID is not found.
"""
@abstractmethod
def add_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Add additional protocols for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
protocols : Sequence[str]
The protocols to add.
"""
@abstractmethod
def set_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Set the protocols for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
protocols : Sequence[str]
The protocols to set.
"""
@abstractmethod
def remove_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
Removes the specified protocols from the peer's protocol list.
Parameters
----------
peer_id : ID
The identifier of the peer from which protocols will be removed.
protocols : Sequence[str]
A sequence of protocol strings to remove.
"""
@abstractmethod
def supports_protocols(self, peer_id: ID, protocols: Sequence[str]) -> list[str]:
"""
Returns the list of protocols from the input sequence that the peer supports.
Parameters
----------
peer_id : ID
The identifier of the peer to check for protocol support.
protocols : Sequence[str]
A sequence of protocol strings to check against the peer's
supported protocols.
"""
@abstractmethod
def first_supported_protocol(self, peer_id: ID, protocols: Sequence[str]) -> str:
"""
Returns the first protocol from the input list that the peer supports.
Parameters
----------
peer_id : ID
The identifier of the peer to check for supported protocols.
protocols : Sequence[str]
A sequence of protocol strings to check.
Returns
-------
str
The first matching protocol string, or an empty string
if none are supported.
"""
@abstractmethod
def clear_protocol_data(self, peer_id: ID) -> None:
"""
Clears all protocol data associated with the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer whose protocol data will be cleared.
"""
# --------PEER-STORE--------
@abstractmethod
def peer_info(self, peer_id: ID) -> PeerInfo:
"""
Retrieve the peer information for the specified peer.
Parameters
----------
peer_id : ID
The identifier of the peer.
Returns
-------
PeerInfo
The peer information object for the given peer.
"""
@abstractmethod
def peer_ids(self) -> list[ID]:
"""
Retrieve all peer identifiers stored in the peer store.
Returns
-------
list[ID]
A list of all peer IDs in the store.
"""
@abstractmethod
def clear_peerdata(self, peer_id: ID) -> None:
"""clear_peerdata"""
# -------------------------- listener interface.py --------------------------
@ -1550,8 +1128,9 @@ class IHost(ABC):
"""
# FIXME: Replace with correct return type
@abstractmethod
def get_mux(self) -> "Multiselect":
def get_mux(self) -> Any:
"""
Retrieve the muxer instance for the host.
@ -1736,60 +1315,6 @@ class IPeerData(ABC):
"""
@abstractmethod
def remove_protocols(self, protocols: Sequence[str]) -> None:
"""
Removes the specified protocols from this peer's list of supported protocols.
Parameters
----------
protocols : Sequence[str]
A sequence of protocol strings to be removed.
"""
@abstractmethod
def supports_protocols(self, protocols: Sequence[str]) -> list[str]:
"""
Returns the list of protocols from the input sequence that are supported
by this peer.
Parameters
----------
protocols : Sequence[str]
A sequence of protocol strings to check against this peer's supported
protocols.
Returns
-------
list[str]
A list of protocol strings that are supported.
"""
@abstractmethod
def first_supported_protocol(self, protocols: Sequence[str]) -> str:
"""
Returns the first protocol from the input list that this peer supports.
Parameters
----------
protocols : Sequence[str]
A sequence of protocol strings to check for support.
Returns
-------
str
The first matching protocol, or an empty string if none are supported.
"""
@abstractmethod
def clear_protocol_data(self) -> None:
"""
Clears all protocol data associated with this peer.
"""
@abstractmethod
def add_addrs(self, addrs: Sequence[Multiaddr]) -> None:
"""
@ -1799,8 +1324,6 @@ class IPeerData(ABC):
----------
addrs : Sequence[Multiaddr]
A sequence of multiaddresses to add.
ttl: inr
Time to live for the peer record
"""
@ -1859,12 +1382,6 @@ class IPeerData(ABC):
"""
@abstractmethod
def clear_metadata(self) -> None:
"""
Clears all metadata entries associated with this peer.
"""
@abstractmethod
def add_pubkey(self, pubkey: PublicKey) -> None:
"""
@ -1923,45 +1440,6 @@ class IPeerData(ABC):
"""
@abstractmethod
def clear_keydata(self) -> None:
"""
Clears all cryptographic key data associated with this peer,
including both public and private keys.
"""
@abstractmethod
def record_latency(self, new_latency: float) -> None:
"""
Records a new latency measurement using
Exponentially Weighted Moving Average (EWMA).
Parameters
----------
new_latency : float
The new round-trip time (RTT) latency value to incorporate
into the EWMA calculation.
"""
@abstractmethod
def latency_EWMA(self) -> float:
"""
Returns the current EWMA value of the recorded latency.
Returns
-------
float
The current latency estimate based on EWMA.
"""
@abstractmethod
def clear_metrics(self) -> None:
"""
Clears all latency-related metrics and resets the internal state.
"""
@abstractmethod
def update_last_identified(self) -> None:
"""
@ -2162,7 +1640,6 @@ class IMultiselectMuxer(ABC):
"""
@abstractmethod
def get_protocols(self) -> tuple[TProtocol | None, ...]:
"""
Retrieve the protocols for which handlers have been registered.
@ -2173,6 +1650,7 @@ class IMultiselectMuxer(ABC):
A tuple of registered protocol names.
"""
return tuple(self.handlers.keys())
@abstractmethod
async def negotiate(

View File

@ -1,26 +0,0 @@
from collections.abc import (
Callable,
)
from libp2p.abc import (
PeerInfo,
)
TTL: int = 60 * 60 # Time-to-live for discovered peers in seconds
class PeerDiscovery:
def __init__(self) -> None:
self._peer_discovered_handlers: list[Callable[[PeerInfo], None]] = []
def register_peer_discovered_handler(
self, handler: Callable[[PeerInfo], None]
) -> None:
self._peer_discovered_handlers.append(handler)
def emit_peer_discovered(self, peer_info: PeerInfo) -> None:
for handler in self._peer_discovered_handlers:
handler(peer_info)
peerDiscovery = PeerDiscovery()

View File

@ -1,91 +0,0 @@
import logging
import socket
from zeroconf import (
EventLoopBlocked,
ServiceInfo,
Zeroconf,
)
logger = logging.getLogger("libp2p.discovery.mdns.broadcaster")
class PeerBroadcaster:
"""
Broadcasts this peer's presence on the local network using mDNS/zeroconf.
Registers a service with the peer's ID in the TXT record as per libp2p spec.
"""
def __init__(
self,
zeroconf: Zeroconf,
service_type: str,
service_name: str,
peer_id: str,
port: int,
):
self.zeroconf = zeroconf
self.service_type = service_type
self.peer_id = peer_id
self.port = port
self.service_name = service_name
# Get the local IP address
local_ip = self._get_local_ip()
hostname = socket.gethostname()
self.service_info = ServiceInfo(
type_=self.service_type,
name=self.service_name,
port=self.port,
properties={b"id": self.peer_id.encode()},
server=f"{hostname}.local.",
addresses=[socket.inet_aton(local_ip)],
)
def _get_local_ip(self) -> str:
"""Get the local IP address of this machine"""
try:
# Connect to a remote address to determine the local IP
# This doesn't actually send data
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as s:
s.connect(("8.8.8.8", 80))
local_ip = s.getsockname()[0]
return local_ip
except Exception:
# Fallback to localhost if we can't determine the IP
return "127.0.0.1"
def register(self) -> None:
"""Register the peer's mDNS service on the network."""
try:
self.zeroconf.register_service(self.service_info)
logger.debug(f"mDNS service registered: {self.service_name}")
except EventLoopBlocked as e:
logger.warning(
"EventLoopBlocked while registering mDNS '%s': %s", self.service_name, e
)
except Exception as e:
logger.error(
"Unexpected error during mDNS registration for '%s': %r",
self.service_name,
e,
)
def unregister(self) -> None:
"""Unregister the peer's mDNS service from the network."""
try:
self.zeroconf.unregister_service(self.service_info)
logger.debug(f"mDNS service unregistered: {self.service_name}")
except EventLoopBlocked as e:
logger.warning(
"EventLoopBlocked while unregistering mDNS '%s': %s",
self.service_name,
e,
)
except Exception as e:
logger.error(
"Unexpected error during mDNS unregistration for '%s': %r",
self.service_name,
e,
)

View File

@ -1,83 +0,0 @@
import logging
import socket
from zeroconf import (
ServiceBrowser,
ServiceInfo,
ServiceListener,
Zeroconf,
)
from libp2p.abc import IPeerStore, Multiaddr
from libp2p.discovery.events.peerDiscovery import peerDiscovery
from libp2p.peer.id import ID
from libp2p.peer.peerinfo import PeerInfo
logger = logging.getLogger("libp2p.discovery.mdns.listener")
class PeerListener(ServiceListener):
"""mDNS listener — now a true ServiceListener subclass."""
def __init__(
self,
peerstore: IPeerStore,
zeroconf: Zeroconf,
service_type: str,
service_name: str,
) -> None:
self.peerstore = peerstore
self.zeroconf = zeroconf
self.service_type = service_type
self.service_name = service_name
self.discovered_services: dict[str, ID] = {}
self.browser = ServiceBrowser(self.zeroconf, self.service_type, listener=self)
def add_service(self, zc: Zeroconf, type_: str, name: str) -> None:
if name == self.service_name:
return
logger.debug(f"Adding service: {name}")
info = zc.get_service_info(type_, name, timeout=5000)
if not info:
return
peer_info = self._extract_peer_info(info)
if peer_info:
self.discovered_services[name] = peer_info.peer_id
self.peerstore.add_addrs(peer_info.peer_id, peer_info.addrs, 10)
peerDiscovery.emit_peer_discovered(peer_info)
logger.debug(f"Discovered Peer: {peer_info.peer_id}")
def remove_service(self, zc: Zeroconf, type_: str, name: str) -> None:
if name == self.service_name:
return
logger.debug(f"Removing service: {name}")
peer_id = self.discovered_services.pop(name)
self.peerstore.clear_addrs(peer_id)
logger.debug(f"Removed Peer: {peer_id}")
def update_service(self, zc: Zeroconf, type_: str, name: str) -> None:
info = zc.get_service_info(type_, name, timeout=5000)
if not info:
return
peer_info = self._extract_peer_info(info)
if peer_info:
self.peerstore.clear_addrs(peer_info.peer_id)
self.peerstore.add_addrs(peer_info.peer_id, peer_info.addrs, 10)
logger.debug(f"Updated Peer {peer_info.peer_id}")
def _extract_peer_info(self, info: ServiceInfo) -> PeerInfo | None:
try:
addrs = [
Multiaddr(f"/ip4/{socket.inet_ntoa(addr)}/tcp/{info.port}")
for addr in info.addresses
]
pid_bytes = info.properties.get(b"id")
if not pid_bytes:
return None
pid = ID.from_base58(pid_bytes.decode())
return PeerInfo(peer_id=pid, addrs=addrs)
except Exception:
return None
def stop(self) -> None:
self.browser.cancel()

View File

@ -1,73 +0,0 @@
"""
mDNS-based peer discovery for py-libp2p.
Conforms to https://github.com/libp2p/specs/blob/master/discovery/mdns.md
Uses zeroconf for mDNS broadcast/listen. Async operations use trio.
"""
import logging
from zeroconf import (
Zeroconf,
)
from libp2p.abc import (
INetworkService,
)
from .broadcaster import (
PeerBroadcaster,
)
from .listener import (
PeerListener,
)
from .utils import (
stringGen,
)
logger = logging.getLogger("libp2p.discovery.mdns")
SERVICE_TYPE = "_p2p._udp.local."
MCAST_PORT = 5353
MCAST_ADDR = "224.0.0.251"
class MDNSDiscovery:
"""
mDNS-based peer discovery for py-libp2p, using zeroconf.
Conforms to the libp2p mDNS discovery spec.
"""
def __init__(self, swarm: INetworkService, port: int = 8000):
self.peer_id = str(swarm.get_peer_id())
self.port = port
self.zeroconf = Zeroconf()
self.serviceName = f"{stringGen()}.{SERVICE_TYPE}"
self.peerstore = swarm.peerstore
self.swarm = swarm
self.broadcaster = PeerBroadcaster(
zeroconf=self.zeroconf,
service_type=SERVICE_TYPE,
service_name=self.serviceName,
peer_id=self.peer_id,
port=self.port,
)
self.listener = PeerListener(
zeroconf=self.zeroconf,
peerstore=self.peerstore,
service_type=SERVICE_TYPE,
service_name=self.serviceName,
)
def start(self) -> None:
"""Register this peer and start listening for others."""
logger.debug(
f"Starting mDNS discovery for peer {self.peer_id} on port {self.port}"
)
self.broadcaster.register()
# Listener is started in constructor
def stop(self) -> None:
"""Unregister this peer and clean up zeroconf resources."""
logger.debug("Stopping mDNS discovery")
self.broadcaster.unregister()
self.zeroconf.close()

View File

@ -1,11 +0,0 @@
import random
import string
def stringGen(len: int = 63) -> str:
"""Generate a random string of lowercase letters and digits."""
charset = string.ascii_lowercase + string.digits
result = []
for _ in range(len):
result.append(random.choice(charset))
return "".join(result)

View File

@ -29,7 +29,6 @@ from libp2p.custom_types import (
StreamHandlerFn,
TProtocol,
)
from libp2p.discovery.mdns.mdns import MDNSDiscovery
from libp2p.host.defaults import (
get_default_protocols,
)
@ -71,7 +70,6 @@ if TYPE_CHECKING:
logger = logging.getLogger("libp2p.network.basic_host")
DEFAULT_NEGOTIATE_TIMEOUT = 5
class BasicHost(IHost):
@ -91,20 +89,15 @@ class BasicHost(IHost):
def __init__(
self,
network: INetworkService,
enable_mDNS: bool = False,
default_protocols: Optional["OrderedDict[TProtocol, StreamHandlerFn]"] = None,
negotitate_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
) -> None:
self._network = network
self._network.set_stream_handler(self._swarm_stream_handler)
self.peerstore = self._network.peerstore
self.negotiate_timeout = negotitate_timeout
# Protocol muxing
default_protocols = default_protocols or get_default_protocols(self)
self.multiselect = Multiselect(dict(default_protocols.items()))
self.multiselect_client = MultiselectClient()
if enable_mDNS:
self.mDNS = MDNSDiscovery(network)
def get_id(self) -> ID:
"""
@ -169,14 +162,7 @@ class BasicHost(IHost):
network = self.get_network()
async with background_trio_service(network):
await network.listen(*listen_addrs)
if hasattr(self, "mDNS") and self.mDNS is not None:
logger.debug("Starting mDNS Discovery")
self.mDNS.start()
try:
yield
finally:
if hasattr(self, "mDNS") and self.mDNS is not None:
self.mDNS.stop()
yield
return _run()
@ -192,10 +178,7 @@ class BasicHost(IHost):
self.multiselect.add_handler(protocol_id, stream_handler)
async def new_stream(
self,
peer_id: ID,
protocol_ids: Sequence[TProtocol],
negotitate_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
self, peer_id: ID, protocol_ids: Sequence[TProtocol]
) -> INetStream:
"""
:param peer_id: peer_id that host is connecting
@ -207,9 +190,7 @@ class BasicHost(IHost):
# Perform protocol muxing to determine protocol to use
try:
selected_protocol = await self.multiselect_client.select_one_of(
list(protocol_ids),
MultiselectCommunicator(net_stream),
negotitate_timeout,
list(protocol_ids), MultiselectCommunicator(net_stream)
)
except MultiselectClientError as error:
logger.debug("fail to open a stream to peer %s, error=%s", peer_id, error)
@ -219,12 +200,7 @@ class BasicHost(IHost):
net_stream.set_protocol(selected_protocol)
return net_stream
async def send_command(
self,
peer_id: ID,
command: str,
response_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
) -> list[str]:
async def send_command(self, peer_id: ID, command: str) -> list[str]:
"""
Send a multistream-select command to the specified peer and return
the response.
@ -238,7 +214,7 @@ class BasicHost(IHost):
try:
response = await self.multiselect_client.query_multistream_command(
MultiselectCommunicator(new_stream), command, response_timeout
MultiselectCommunicator(new_stream), command
)
except MultiselectClientError as error:
logger.debug("fail to open a stream to peer %s, error=%s", peer_id, error)
@ -277,7 +253,7 @@ class BasicHost(IHost):
# Perform protocol muxing to determine protocol to use
try:
protocol, handler = await self.multiselect.negotiate(
MultiselectCommunicator(net_stream), self.negotiate_timeout
MultiselectCommunicator(net_stream)
)
except MultiselectError as error:
peer_id = net_stream.muxed_conn.peer_id

View File

@ -26,8 +26,5 @@ if TYPE_CHECKING:
def get_default_protocols(host: IHost) -> "OrderedDict[TProtocol, StreamHandlerFn]":
return OrderedDict(
(
(IdentifyID, identify_handler_for(host, use_varint_format=True)),
(PingID, handle_ping),
)
((IdentifyID, identify_handler_for(host)), (PingID, handle_ping))
)

View File

@ -18,10 +18,8 @@ from libp2p.peer.peerinfo import (
class RoutedHost(BasicHost):
_router: IPeerRouting
def __init__(
self, network: INetworkService, router: IPeerRouting, enable_mDNS: bool = False
):
super().__init__(network, enable_mDNS)
def __init__(self, network: INetworkService, router: IPeerRouting):
super().__init__(network)
self._router = router
async def connect(self, peer_info: PeerInfo) -> None:

View File

@ -16,9 +16,7 @@ from libp2p.network.stream.exceptions import (
StreamClosed,
)
from libp2p.utils import (
decode_varint_with_size,
get_agent_version,
varint,
)
from .pb.identify_pb2 import (
@ -61,7 +59,7 @@ def _mk_identify_protobuf(
) -> Identify:
public_key = host.get_public_key()
laddrs = host.get_addrs()
protocols = tuple(str(p) for p in host.get_mux().get_protocols() if p is not None)
protocols = host.get_mux().get_protocols()
observed_addr = observed_multiaddr.to_bytes() if observed_multiaddr else b""
return Identify(
@ -74,47 +72,7 @@ def _mk_identify_protobuf(
)
def parse_identify_response(response: bytes) -> Identify:
"""
Parse identify response that could be either:
- Old format: raw protobuf
- New format: length-prefixed protobuf
This function provides backward and forward compatibility.
"""
# Try new format first: length-prefixed protobuf
if len(response) >= 1:
length, varint_size = decode_varint_with_size(response)
if varint_size > 0 and length > 0 and varint_size + length <= len(response):
protobuf_data = response[varint_size : varint_size + length]
try:
identify_response = Identify()
identify_response.ParseFromString(protobuf_data)
# Sanity check: must have agent_version (protocol_version is optional)
if identify_response.agent_version:
logger.debug(
"Parsed length-prefixed identify response (new format)"
)
return identify_response
except Exception:
pass # Fall through to old format
# Fall back to old format: raw protobuf
try:
identify_response = Identify()
identify_response.ParseFromString(response)
logger.debug("Parsed raw protobuf identify response (old format)")
return identify_response
except Exception as e:
logger.error(f"Failed to parse identify response: {e}")
logger.error(f"Response length: {len(response)}")
logger.error(f"Response hex: {response.hex()}")
raise
def identify_handler_for(
host: IHost, use_varint_format: bool = False
) -> StreamHandlerFn:
def identify_handler_for(host: IHost) -> StreamHandlerFn:
async def handle_identify(stream: INetStream) -> None:
# get observed address from ``stream``
peer_id = (
@ -142,21 +100,7 @@ def identify_handler_for(
response = protobuf.SerializeToString()
try:
if use_varint_format:
# Send length-prefixed protobuf message (new format)
await stream.write(varint.encode_uvarint(len(response)))
await stream.write(response)
logger.debug(
"Sent new format (length-prefixed) identify response to %s",
peer_id,
)
else:
# Send raw protobuf message (old format for backward compatibility)
await stream.write(response)
logger.debug(
"Sent old format (raw protobuf) identify response to %s",
peer_id,
)
await stream.write(response)
except StreamClosed:
logger.debug("Fail to respond to %s request: stream closed", ID)
else:

View File

@ -25,10 +25,6 @@ from libp2p.peer.id import (
)
from libp2p.utils import (
get_agent_version,
varint,
)
from libp2p.utils.varint import (
decode_varint_from_bytes,
)
from ..identify.identify import (
@ -44,72 +40,22 @@ logger = logging.getLogger(__name__)
ID_PUSH = TProtocol("/ipfs/id/push/1.0.0")
PROTOCOL_VERSION = "ipfs/0.1.0"
AGENT_VERSION = get_agent_version()
CONCURRENCY_LIMIT = 10
def identify_push_handler_for(
host: IHost, use_varint_format: bool = True
) -> StreamHandlerFn:
def identify_push_handler_for(host: IHost) -> StreamHandlerFn:
"""
Create a handler for the identify/push protocol.
This handler receives pushed identify messages from remote peers and updates
the local peerstore with the new information.
Args:
host: The libp2p host.
use_varint_format: True=length-prefixed, False=raw protobuf.
"""
async def handle_identify_push(stream: INetStream) -> None:
peer_id = stream.muxed_conn.peer_id
try:
if use_varint_format:
# Read length-prefixed identify message from the stream
# First read the varint length prefix
length_bytes = b""
while True:
b = await stream.read(1)
if not b:
break
length_bytes += b
if b[0] & 0x80 == 0:
break
if not length_bytes:
logger.warning("No length prefix received from peer %s", peer_id)
return
msg_length = decode_varint_from_bytes(length_bytes)
# Read the protobuf message
data = await stream.read(msg_length)
if len(data) != msg_length:
logger.warning("Incomplete message received from peer %s", peer_id)
return
else:
# Read raw protobuf message from the stream
# For raw format, we need to read all data before the stream is closed
data = b""
try:
# Read all available data in a single operation
data = await stream.read()
except StreamClosed:
# Try to read any remaining data
try:
data = await stream.read()
except Exception:
pass
# If we got no data, log a warning and return
if not data:
logger.warning(
"No data received in raw format from peer %s", peer_id
)
return
# Read the identify message from the stream
data = await stream.read()
identify_msg = Identify()
identify_msg.ParseFromString(data)
@ -186,11 +132,7 @@ async def _update_peerstore_from_identify(
async def push_identify_to_peer(
host: IHost,
peer_id: ID,
observed_multiaddr: Multiaddr | None = None,
limit: trio.Semaphore = trio.Semaphore(CONCURRENCY_LIMIT),
use_varint_format: bool = True,
host: IHost, peer_id: ID, observed_multiaddr: Multiaddr | None = None
) -> bool:
"""
Push an identify message to a specific peer.
@ -198,78 +140,52 @@ async def push_identify_to_peer(
This function opens a stream to the peer using the identify/push protocol,
sends the identify message, and closes the stream.
Args:
host: The libp2p host.
peer_id: The peer ID to push to.
observed_multiaddr: The observed multiaddress (optional).
limit: Semaphore for concurrency control.
use_varint_format: True=length-prefixed, False=raw protobuf.
Returns:
bool: True if the push was successful, False otherwise.
Returns
-------
bool
True if the push was successful, False otherwise.
"""
async with limit:
try:
# Create a new stream to the peer using the identify/push protocol
stream = await host.new_stream(peer_id, [ID_PUSH])
try:
# Create a new stream to the peer using the identify/push protocol
stream = await host.new_stream(peer_id, [ID_PUSH])
# Create the identify message
identify_msg = _mk_identify_protobuf(host, observed_multiaddr)
response = identify_msg.SerializeToString()
# Create the identify message
identify_msg = _mk_identify_protobuf(host, observed_multiaddr)
response = identify_msg.SerializeToString()
if use_varint_format:
# Send length-prefixed identify message
await stream.write(varint.encode_uvarint(len(response)))
await stream.write(response)
else:
# Send raw protobuf message
await stream.write(response)
# Send the identify message
await stream.write(response)
# Close the stream
await stream.close()
# Close the stream
await stream.close()
logger.debug("Successfully pushed identify to peer %s", peer_id)
return True
except Exception as e:
logger.error("Error pushing identify to peer %s: %s", peer_id, e)
return False
logger.debug("Successfully pushed identify to peer %s", peer_id)
return True
except Exception as e:
logger.error("Error pushing identify to peer %s: %s", peer_id, e)
return False
async def push_identify_to_peers(
host: IHost,
peer_ids: set[ID] | None = None,
observed_multiaddr: Multiaddr | None = None,
use_varint_format: bool = True,
) -> None:
"""
Push an identify message to multiple peers in parallel.
If peer_ids is None, push to all connected peers.
Args:
host: The libp2p host.
peer_ids: Set of peer IDs to push to (if None, push to all connected peers).
observed_multiaddr: The observed multiaddress (optional).
use_varint_format: True=length-prefixed, False=raw protobuf.
"""
if peer_ids is None:
# Get all connected peers
peer_ids = set(host.get_connected_peers())
# Create a single shared semaphore for concurrency control
limit = trio.Semaphore(CONCURRENCY_LIMIT)
peer_ids = set(host.get_peerstore().peer_ids())
# Push to each peer in parallel using a trio.Nursery
# limiting concurrent connections to CONCURRENCY_LIMIT
# TODO: Consider using a bounded nursery to limit concurrency
# and avoid overwhelming the network. This can be done by using
# trio.open_nursery(max_concurrent=10) or similar.
# For now, we will use an unbounded nursery for simplicity.
async with trio.open_nursery() as nursery:
for peer_id in peer_ids:
nursery.start_soon(
push_identify_to_peer,
host,
peer_id,
observed_multiaddr,
limit,
use_varint_format,
)
nursery.start_soon(push_identify_to_peer, host, peer_id, observed_multiaddr)

View File

@ -18,13 +18,6 @@ from libp2p.crypto.keys import (
PublicKey,
)
"""
Latency EWMA Smoothing governs the deacy of the EWMA (the speed at which
is changes). This must be a normalized (0-1) value.
1 is 100% change, 0 is no change.
"""
LATENCY_EWMA_SMOOTHING = 0.1
class PeerData(IPeerData):
pubkey: PublicKey | None
@ -34,7 +27,6 @@ class PeerData(IPeerData):
addrs: list[Multiaddr]
last_identified: int
ttl: int # Keep ttl=0 by default for always valid
latmap: float
def __init__(self) -> None:
self.pubkey = None
@ -44,9 +36,6 @@ class PeerData(IPeerData):
self.addrs = []
self.last_identified = int(time.time())
self.ttl = 0
self.latmap = 0
# --------PROTO-BOOK--------
def get_protocols(self) -> list[str]:
"""
@ -66,37 +55,6 @@ class PeerData(IPeerData):
"""
self.protocols = list(protocols)
def remove_protocols(self, protocols: Sequence[str]) -> None:
"""
:param protocols: protocols to remove
"""
for protocol in protocols:
if protocol in self.protocols:
self.protocols.remove(protocol)
def supports_protocols(self, protocols: Sequence[str]) -> list[str]:
"""
:param protocols: protocols to check from
:return: all supported protocols in the given list
"""
return [proto for proto in protocols if proto in self.protocols]
def first_supported_protocol(self, protocols: Sequence[str]) -> str:
"""
:param protocols: protocols to check from
:return: first supported protocol in the given list
"""
for protocol in protocols:
if protocol in self.protocols:
return protocol
return "None supported"
def clear_protocol_data(self) -> None:
"""Clear all protocols"""
self.protocols = []
# -------ADDR-BOOK---------
def add_addrs(self, addrs: Sequence[Multiaddr]) -> None:
"""
:param addrs: multiaddresses to add
@ -115,7 +73,6 @@ class PeerData(IPeerData):
"""Clear all addresses."""
self.addrs = []
# -------METADATA-----------
def put_metadata(self, key: str, val: Any) -> None:
"""
:param key: key in KV pair
@ -133,11 +90,6 @@ class PeerData(IPeerData):
return self.metadata[key]
raise PeerDataError("key not found")
def clear_metadata(self) -> None:
"""Clears metadata."""
self.metadata = {}
# -------KEY-BOOK---------------
def add_pubkey(self, pubkey: PublicKey) -> None:
"""
:param pubkey:
@ -168,41 +120,9 @@ class PeerData(IPeerData):
raise PeerDataError("private key not found")
return self.privkey
def clear_keydata(self) -> None:
"""Clears keydata"""
self.pubkey = None
self.privkey = None
# ----------METRICS--------------
def record_latency(self, new_latency: float) -> None:
"""
Records a new latency measurement for the given peer
using Exponentially Weighted Moving Average (EWMA)
:param new_latency: the new latency value
"""
s = LATENCY_EWMA_SMOOTHING
if s > 1 or s < 0:
s = 0.1
if self.latmap == 0:
self.latmap = new_latency
else:
prev = self.latmap
updated = ((1.0 - s) * prev) + (s * new_latency)
self.latmap = updated
def latency_EWMA(self) -> float:
"""Returns the latency EWMA value"""
return self.latmap
def clear_metrics(self) -> None:
"""Clear the latency metrics"""
self.latmap = 0
def update_last_identified(self) -> None:
self.last_identified = int(time.time())
# ----------TTL------------------
def get_last_identified(self) -> int:
"""
:return: last identified timestamp

View File

@ -3,11 +3,9 @@ from collections.abc import (
)
from typing import (
Any,
cast,
)
import multiaddr
from multiaddr.protocols import Protocol
from .id import (
ID,
@ -44,8 +42,7 @@ def info_from_p2p_addr(addr: multiaddr.Multiaddr) -> PeerInfo:
p2p_protocols = p2p_part.protocols()
if not p2p_protocols:
raise InvalidAddrError("The last part of the address has no protocols")
last_protocol = cast(Protocol, p2p_part.protocols()[0])
last_protocol = p2p_protocols[0]
if last_protocol is None:
raise InvalidAddrError("The last protocol is None")

View File

@ -2,7 +2,6 @@ from collections import (
defaultdict,
)
from collections.abc import (
AsyncIterable,
Sequence,
)
from typing import (
@ -12,8 +11,6 @@ from typing import (
from multiaddr import (
Multiaddr,
)
import trio
from trio import MemoryReceiveChannel, MemorySendChannel
from libp2p.abc import (
IPeerStore,
@ -43,7 +40,6 @@ class PeerStore(IPeerStore):
def __init__(self) -> None:
self.peer_data_map = defaultdict(PeerData)
self.addr_update_channels: dict[ID, MemorySendChannel[Multiaddr]] = {}
def peer_info(self, peer_id: ID) -> PeerInfo:
"""
@ -57,33 +53,6 @@ class PeerStore(IPeerStore):
return PeerInfo(peer_id, peer_data.get_addrs())
raise PeerStoreError("peer ID not found")
def peer_ids(self) -> list[ID]:
"""
:return: all of the peer IDs stored in peer store
"""
return list(self.peer_data_map.keys())
def clear_peerdata(self, peer_id: ID) -> None:
"""Clears all data associated with the given peer_id."""
if peer_id in self.peer_data_map:
del self.peer_data_map[peer_id]
else:
raise PeerStoreError("peer ID not found")
def valid_peer_ids(self) -> list[ID]:
"""
:return: all of the valid peer IDs stored in peer store
"""
valid_peer_ids: list[ID] = []
for peer_id, peer_data in self.peer_data_map.items():
if not peer_data.is_expired():
valid_peer_ids.append(peer_id)
else:
peer_data.clear_addrs()
return valid_peer_ids
# --------PROTO-BOOK--------
def get_protocols(self, peer_id: ID) -> list[str]:
"""
:param peer_id: peer ID to get protocols for
@ -110,31 +79,23 @@ class PeerStore(IPeerStore):
peer_data = self.peer_data_map[peer_id]
peer_data.set_protocols(list(protocols))
def remove_protocols(self, peer_id: ID, protocols: Sequence[str]) -> None:
"""
:param peer_id: peer ID to get info for
:param protocols: unsupported protocols to remove
"""
peer_data = self.peer_data_map[peer_id]
peer_data.remove_protocols(protocols)
def supports_protocols(self, peer_id: ID, protocols: Sequence[str]) -> list[str]:
def peer_ids(self) -> list[ID]:
"""
:return: all of the peer IDs stored in peer store
"""
peer_data = self.peer_data_map[peer_id]
return peer_data.supports_protocols(protocols)
return list(self.peer_data_map.keys())
def first_supported_protocol(self, peer_id: ID, protocols: Sequence[str]) -> str:
peer_data = self.peer_data_map[peer_id]
return peer_data.first_supported_protocol(protocols)
def clear_protocol_data(self, peer_id: ID) -> None:
"""Clears prtocoldata"""
peer_data = self.peer_data_map[peer_id]
peer_data.clear_protocol_data()
# ------METADATA---------
def valid_peer_ids(self) -> list[ID]:
"""
:return: all of the valid peer IDs stored in peer store
"""
valid_peer_ids: list[ID] = []
for peer_id, peer_data in self.peer_data_map.items():
if not peer_data.is_expired():
valid_peer_ids.append(peer_id)
else:
peer_data.clear_addrs()
return valid_peer_ids
def get(self, peer_id: ID, key: str) -> Any:
"""
@ -160,13 +121,6 @@ class PeerStore(IPeerStore):
peer_data = self.peer_data_map[peer_id]
peer_data.put_metadata(key, val)
def clear_metadata(self, peer_id: ID) -> None:
"""Clears metadata"""
peer_data = self.peer_data_map[peer_id]
peer_data.clear_metadata()
# -------ADDR-BOOK--------
def add_addr(self, peer_id: ID, addr: Multiaddr, ttl: int = 0) -> None:
"""
:param peer_id: peer ID to add address for
@ -186,13 +140,6 @@ class PeerStore(IPeerStore):
peer_data.set_ttl(ttl)
peer_data.update_last_identified()
if peer_id in self.addr_update_channels:
for addr in addrs:
try:
self.addr_update_channels[peer_id].send_nowait(addr)
except trio.WouldBlock:
pass # Or consider logging / dropping / replacing stream
def addrs(self, peer_id: ID) -> list[Multiaddr]:
"""
:param peer_id: peer ID to get addrs for
@ -218,7 +165,7 @@ class PeerStore(IPeerStore):
def peers_with_addrs(self) -> list[ID]:
"""
:return: all of the peer IDs which has addrsfloat stored in peer store
:return: all of the peer IDs which has addrs stored in peer store
"""
# Add all peers with addrs at least 1 to output
output: list[ID] = []
@ -232,27 +179,6 @@ class PeerStore(IPeerStore):
peer_data.clear_addrs()
return output
async def addr_stream(self, peer_id: ID) -> AsyncIterable[Multiaddr]:
"""
Returns an async stream of newly added addresses for the given peer.
This function allows consumers to subscribe to address updates for a peer
and receive each new address as it is added via `add_addr` or `add_addrs`.
:param peer_id: The ID of the peer to monitor address updates for.
:return: An async iterator yielding Multiaddr instances as they are added.
"""
send: MemorySendChannel[Multiaddr]
receive: MemoryReceiveChannel[Multiaddr]
send, receive = trio.open_memory_channel(0)
self.addr_update_channels[peer_id] = send
async for addr in receive:
yield addr
# -------KEY-BOOK---------
def add_pubkey(self, peer_id: ID, pubkey: PublicKey) -> None:
"""
:param peer_id: peer ID to add public key for
@ -313,45 +239,6 @@ class PeerStore(IPeerStore):
self.add_pubkey(peer_id, key_pair.public_key)
self.add_privkey(peer_id, key_pair.private_key)
def peer_with_keys(self) -> list[ID]:
"""Returns the peer_ids for which keys are stored"""
return [
peer_id
for peer_id, pdata in self.peer_data_map.items()
if pdata.pubkey is not None
]
def clear_keydata(self, peer_id: ID) -> None:
"""Clears the keys of the peer"""
peer_data = self.peer_data_map[peer_id]
peer_data.clear_keydata()
# --------METRICS--------
def record_latency(self, peer_id: ID, RTT: float) -> None:
"""
Records a new latency measurement for the given peer
using Exponentially Weighted Moving Average (EWMA)
:param peer_id: peer ID to get private key for
:param RTT: the new latency value (round trip time)
"""
peer_data = self.peer_data_map[peer_id]
peer_data.record_latency(RTT)
def latency_EWMA(self, peer_id: ID) -> float:
"""
:param peer_id: peer ID to get private key for
:return: The latency EWMA value for that peer
"""
peer_data = self.peer_data_map[peer_id]
return peer_data.latency_EWMA()
def clear_metrics(self, peer_id: ID) -> None:
"""Clear the latency metrics"""
peer_data = self.peer_data_map[peer_id]
peer_data.clear_metrics()
class PeerStoreError(KeyError):
"""Raised when peer ID is not found in peer store."""

View File

@ -1,5 +1,3 @@
import trio
from libp2p.abc import (
IMultiselectCommunicator,
IMultiselectMuxer,
@ -16,7 +14,6 @@ from .exceptions import (
MULTISELECT_PROTOCOL_ID = "/multistream/1.0.0"
PROTOCOL_NOT_FOUND_MSG = "na"
DEFAULT_NEGOTIATE_TIMEOUT = 5
class Multiselect(IMultiselectMuxer):
@ -50,68 +47,47 @@ class Multiselect(IMultiselectMuxer):
# FIXME: Make TProtocol Optional[TProtocol] to keep types consistent
async def negotiate(
self,
communicator: IMultiselectCommunicator,
negotiate_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
self, communicator: IMultiselectCommunicator
) -> tuple[TProtocol, StreamHandlerFn | None]:
"""
Negotiate performs protocol selection.
:param stream: stream to negotiate on
:param negotiate_timeout: timeout for negotiation
:return: selected protocol name, handler function
:raise MultiselectError: raised when negotiation failed
"""
try:
with trio.fail_after(negotiate_timeout):
await self.handshake(communicator)
await self.handshake(communicator)
while True:
while True:
try:
command = await communicator.read()
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
if command == "ls":
supported_protocols = [p for p in self.handlers.keys() if p is not None]
response = "\n".join(supported_protocols) + "\n"
try:
await communicator.write(response)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
else:
protocol = TProtocol(command)
if protocol in self.handlers:
try:
command = await communicator.read()
await communicator.write(protocol)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
if command == "ls":
supported_protocols = [
p for p in self.handlers.keys() if p is not None
]
response = "\n".join(supported_protocols) + "\n"
return protocol, self.handlers[protocol]
try:
await communicator.write(PROTOCOL_NOT_FOUND_MSG)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
try:
await communicator.write(response)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
else:
protocol = TProtocol(command)
if protocol in self.handlers:
try:
await communicator.write(protocol)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
return protocol, self.handlers[protocol]
try:
await communicator.write(PROTOCOL_NOT_FOUND_MSG)
except MultiselectCommunicatorError as error:
raise MultiselectError() from error
raise MultiselectError("Negotiation failed: no matching protocol")
except trio.TooSlowError:
raise MultiselectError("handshake read timeout")
def get_protocols(self) -> tuple[TProtocol | None, ...]:
"""
Retrieve the protocols for which handlers have been registered.
Returns
-------
tuple[TProtocol, ...]
A tuple of registered protocol names.
"""
return tuple(self.handlers.keys())
raise MultiselectError("Negotiation failed: no matching protocol")
async def handshake(self, communicator: IMultiselectCommunicator) -> None:
"""

View File

@ -2,8 +2,6 @@ from collections.abc import (
Sequence,
)
import trio
from libp2p.abc import (
IMultiselectClient,
IMultiselectCommunicator,
@ -19,7 +17,6 @@ from .exceptions import (
MULTISELECT_PROTOCOL_ID = "/multistream/1.0.0"
PROTOCOL_NOT_FOUND_MSG = "na"
DEFAULT_NEGOTIATE_TIMEOUT = 5
class MultiselectClient(IMultiselectClient):
@ -43,7 +40,6 @@ class MultiselectClient(IMultiselectClient):
try:
handshake_contents = await communicator.read()
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error
@ -51,10 +47,7 @@ class MultiselectClient(IMultiselectClient):
raise MultiselectClientError("multiselect protocol ID mismatch")
async def select_one_of(
self,
protocols: Sequence[TProtocol],
communicator: IMultiselectCommunicator,
negotitate_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
self, protocols: Sequence[TProtocol], communicator: IMultiselectCommunicator
) -> TProtocol:
"""
For each protocol, send message to multiselect selecting protocol and
@ -63,32 +56,22 @@ class MultiselectClient(IMultiselectClient):
:param protocol: protocol to select
:param communicator: communicator to use to communicate with counterparty
:param negotiate_timeout: timeout for negotiation
:return: selected protocol
:raise MultiselectClientError: raised when protocol negotiation failed
"""
try:
with trio.fail_after(negotitate_timeout):
await self.handshake(communicator)
await self.handshake(communicator)
for protocol in protocols:
try:
selected_protocol = await self.try_select(
communicator, protocol
)
return selected_protocol
except MultiselectClientError:
pass
for protocol in protocols:
try:
selected_protocol = await self.try_select(communicator, protocol)
return selected_protocol
except MultiselectClientError:
pass
raise MultiselectClientError("protocols not supported")
except trio.TooSlowError:
raise MultiselectClientError("response timed out")
raise MultiselectClientError("protocols not supported")
async def query_multistream_command(
self,
communicator: IMultiselectCommunicator,
command: str,
response_timeout: int = DEFAULT_NEGOTIATE_TIMEOUT,
self, communicator: IMultiselectCommunicator, command: str
) -> list[str]:
"""
Send a multistream-select command over the given communicator and return
@ -96,32 +79,26 @@ class MultiselectClient(IMultiselectClient):
:param communicator: communicator to use to communicate with counterparty
:param command: supported multistream-select command(e.g., ls)
:param negotiate_timeout: timeout for negotiation
:raise MultiselectClientError: If the communicator fails to process data.
:return: list of strings representing the response from peer.
"""
await self.handshake(communicator)
if command == "ls":
try:
await communicator.write("ls")
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error
else:
raise ValueError("Command not supported")
try:
with trio.fail_after(response_timeout):
await self.handshake(communicator)
response = await communicator.read()
response_list = response.strip().splitlines()
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error
if command == "ls":
try:
await communicator.write("ls")
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error
else:
raise ValueError("Command not supported")
try:
response = await communicator.read()
response_list = response.strip().splitlines()
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error
return response_list
except trio.TooSlowError:
raise MultiselectClientError("command response timed out")
return response_list
async def try_select(
self, communicator: IMultiselectCommunicator, protocol: TProtocol
@ -141,7 +118,6 @@ class MultiselectClient(IMultiselectClient):
try:
response = await communicator.read()
except MultiselectCommunicatorError as error:
raise MultiselectClientError() from error

View File

@ -12,9 +12,15 @@ from libp2p.abc import (
from libp2p.custom_types import (
TProtocol,
)
from libp2p.network.stream.exceptions import (
StreamClosed,
)
from libp2p.peer.id import (
ID,
)
from libp2p.utils import (
encode_varint_prefixed,
)
from .exceptions import (
PubsubRouterError,
@ -114,7 +120,13 @@ class FloodSub(IPubsubRouter):
if peer_id not in pubsub.peers:
continue
stream = pubsub.peers[peer_id]
await pubsub.write_msg(stream, rpc_msg)
# FIXME: We should add a `WriteMsg` similar to write delimited messages.
# Ref: https://github.com/libp2p/go-libp2p-pubsub/blob/master/comm.go#L107
try:
await stream.write(encode_varint_prefixed(rpc_msg.SerializeToString()))
except StreamClosed:
logger.debug("Fail to publish message to %s: stream closed", peer_id)
pubsub._handle_dead_peer(peer_id)
async def join(self, topic: str) -> None:
"""

View File

@ -24,6 +24,9 @@ from libp2p.abc import (
from libp2p.custom_types import (
TProtocol,
)
from libp2p.network.stream.exceptions import (
StreamClosed,
)
from libp2p.peer.id import (
ID,
)
@ -41,6 +44,9 @@ from libp2p.pubsub import (
from libp2p.tools.async_service import (
Service,
)
from libp2p.utils import (
encode_varint_prefixed,
)
from .exceptions import (
NoPubsubAttached,
@ -261,10 +267,13 @@ class GossipSub(IPubsubRouter, Service):
if peer_id not in self.pubsub.peers:
continue
stream = self.pubsub.peers[peer_id]
# TODO: Go use `sendRPC`, which possibly piggybacks gossip/control messages.
await self.pubsub.write_msg(stream, rpc_msg)
# FIXME: We should add a `WriteMsg` similar to write delimited messages.
# Ref: https://github.com/libp2p/go-libp2p-pubsub/blob/master/comm.go#L107
try:
await stream.write(encode_varint_prefixed(rpc_msg.SerializeToString()))
except StreamClosed:
logger.debug("Fail to publish message to %s: stream closed", peer_id)
self.pubsub._handle_dead_peer(peer_id)
for topic in pubsub_msg.topicIDs:
self.time_since_last_publish[topic] = int(time.time())
@ -820,6 +829,8 @@ class GossipSub(IPubsubRouter, Service):
packet.publish.extend(msgs_to_forward)
# 2) Serialize that packet
rpc_msg: bytes = packet.SerializeToString()
if self.pubsub is None:
raise NoPubsubAttached
@ -833,7 +844,14 @@ class GossipSub(IPubsubRouter, Service):
peer_stream = self.pubsub.peers[sender_peer_id]
# 4) And write the packet to the stream
await self.pubsub.write_msg(peer_stream, packet)
try:
await peer_stream.write(encode_varint_prefixed(rpc_msg))
except StreamClosed:
logger.debug(
"Fail to responed to iwant request from %s: stream closed",
sender_peer_id,
)
self.pubsub._handle_dead_peer(sender_peer_id)
async def handle_graft(
self, graft_msg: rpc_pb2.ControlGraft, sender_peer_id: ID
@ -975,6 +993,8 @@ class GossipSub(IPubsubRouter, Service):
packet: rpc_pb2.RPC = rpc_pb2.RPC()
packet.control.CopyFrom(control_msg)
rpc_msg: bytes = packet.SerializeToString()
# Get stream for peer from pubsub
if to_peer not in self.pubsub.peers:
logger.debug(
@ -984,4 +1004,8 @@ class GossipSub(IPubsubRouter, Service):
peer_stream = self.pubsub.peers[to_peer]
# Write rpc to stream
await self.pubsub.write_msg(peer_stream, packet)
try:
await peer_stream.write(encode_varint_prefixed(rpc_msg))
except StreamClosed:
logger.debug("Fail to emit control message to %s: stream closed", to_peer)
self.pubsub._handle_dead_peer(to_peer)

View File

@ -11,6 +11,10 @@ import functools
import hashlib
import logging
import time
from typing import (
NamedTuple,
cast,
)
import base58
import trio
@ -26,6 +30,8 @@ from libp2p.crypto.keys import (
PrivateKey,
)
from libp2p.custom_types import (
AsyncValidatorFn,
SyncValidatorFn,
TProtocol,
ValidatorFn,
)
@ -60,7 +66,6 @@ from libp2p.utils import (
encode_varint_prefixed,
read_varint_prefixed_bytes,
)
from libp2p.utils.varint import encode_uvarint
from .pb import (
rpc_pb2,
@ -71,11 +76,6 @@ from .pubsub_notifee import (
from .subscription import (
TrioSubscriptionAPI,
)
from .validation_throttler import (
TopicValidator,
ValidationResult,
ValidationThrottler,
)
from .validators import (
PUBSUB_SIGNING_PREFIX,
signature_validator,
@ -96,6 +96,11 @@ def get_content_addressed_msg_id(msg: rpc_pb2.Message) -> bytes:
return base64.b64encode(hashlib.sha256(msg.data).digest())
class TopicValidator(NamedTuple):
validator: ValidatorFn
is_async: bool
class Pubsub(Service, IPubsub):
host: IHost
@ -137,11 +142,6 @@ class Pubsub(Service, IPubsub):
msg_id_constructor: Callable[
[rpc_pb2.Message], bytes
] = get_peer_and_seqno_msg_id,
# TODO: these values have been copied from Go, but try to tune these dynamically
validation_queue_size: int = 32,
global_throttle_limit: int = 8192,
default_topic_throttle_limit: int = 1024,
validation_worker_count: int | None = None,
) -> None:
"""
Construct a new Pubsub object, which is responsible for handling all
@ -202,15 +202,7 @@ class Pubsub(Service, IPubsub):
# Create peers map, which maps peer_id (as string) to stream (to a given peer)
self.peers = {}
# Validation Throttler
self.validation_throttler = ValidationThrottler(
queue_size=validation_queue_size,
global_throttle_limit=global_throttle_limit,
default_topic_throttle_limit=default_topic_throttle_limit,
worker_count=validation_worker_count or 4,
)
# Keep a mapping of topic -> TopicValidator for easier lookup
# Map of topic to topic validator
self.topic_validators = {}
self.counter = int(time.time())
@ -222,19 +214,10 @@ class Pubsub(Service, IPubsub):
self.event_handle_dead_peer_queue_started = trio.Event()
async def run(self) -> None:
self.manager.run_daemon_task(self._start_validation_throttler)
self.manager.run_daemon_task(self.handle_peer_queue)
self.manager.run_daemon_task(self.handle_dead_peer_queue)
await self.manager.wait_finished()
async def _start_validation_throttler(self) -> None:
"""Start validation throttler in current nursery context"""
async with trio.open_nursery() as nursery:
await self.validation_throttler.start(nursery)
# Keep nursery alive until service stops
while self.manager.is_running:
await self.manager.wait_finished()
@property
def my_id(self) -> ID:
return self.host.get_id()
@ -314,12 +297,7 @@ class Pubsub(Service, IPubsub):
)
def set_topic_validator(
self,
topic: str,
validator: ValidatorFn,
is_async_validator: bool,
timeout: float | None = None,
throttle_limit: int | None = None,
self, topic: str, validator: ValidatorFn, is_async_validator: bool
) -> None:
"""
Register a validator under the given topic. One topic can only have one
@ -328,18 +306,8 @@ class Pubsub(Service, IPubsub):
:param topic: the topic to register validator under
:param validator: the validator used to validate messages published to the topic
:param is_async_validator: indicate if the validator is an asynchronous validator
:param timeout: optional timeout for the validator
:param throttle_limit: optional throttle limit for the validator
""" # noqa: E501
# Create throttled topic validator
topic_validator = self.validation_throttler.create_topic_validator(
topic=topic,
validator=validator,
is_async=is_async_validator,
timeout=timeout,
throttle_limit=throttle_limit,
)
self.topic_validators[topic] = topic_validator
self.topic_validators[topic] = TopicValidator(validator, is_async_validator)
def remove_topic_validator(self, topic: str) -> None:
"""
@ -349,18 +317,17 @@ class Pubsub(Service, IPubsub):
"""
self.topic_validators.pop(topic, None)
def get_msg_validators(self, msg: rpc_pb2.Message) -> list[TopicValidator]:
def get_msg_validators(self, msg: rpc_pb2.Message) -> tuple[TopicValidator, ...]:
"""
Get all validators corresponding to the topics in the message.
:param msg: the message published to the topic
:return: list of topic validators for the message's topics
"""
return [
return tuple(
self.topic_validators[topic]
for topic in msg.topicIDs
if topic in self.topic_validators
]
)
def add_to_blacklist(self, peer_id: ID) -> None:
"""
@ -696,56 +663,38 @@ class Pubsub(Service, IPubsub):
:param msg_forwarder: the peer who forward us the message.
:param msg: the message.
"""
# Get applicable validators for this message
validators = self.get_msg_validators(msg)
sync_topic_validators: list[SyncValidatorFn] = []
async_topic_validators: list[AsyncValidatorFn] = []
for topic_validator in self.get_msg_validators(msg):
if topic_validator.is_async:
async_topic_validators.append(
cast(AsyncValidatorFn, topic_validator.validator)
)
else:
sync_topic_validators.append(
cast(SyncValidatorFn, topic_validator.validator)
)
if not validators:
# No validators, accept immediately
return
for validator in sync_topic_validators:
if not validator(msg_forwarder, msg):
raise ValidationError(f"Validation failed for msg={msg}")
# Use trio.Event for async coordination
validation_event = trio.Event()
result_container: dict[str, ValidationResult | None | Exception] = {
"result": None,
"error": None,
}
# TODO: Implement throttle on async validators
def handle_validation_result(
result: ValidationResult, error: Exception | None
) -> None:
result_container["result"] = result
result_container["error"] = error
validation_event.set()
if len(async_topic_validators) > 0:
# Appends to lists are thread safe in CPython
results = []
# Submit for throttled validation
success = await self.validation_throttler.submit_validation(
validators=validators,
msg_forwarder=msg_forwarder,
msg=msg,
result_callback=handle_validation_result,
)
async def run_async_validator(func: AsyncValidatorFn) -> None:
result = await func(msg_forwarder, msg)
results.append(result)
if not success:
# Validation was throttled at queue level
raise ValidationError("Validation throttled at queue level")
async with trio.open_nursery() as nursery:
for async_validator in async_topic_validators:
nursery.start_soon(run_async_validator, async_validator)
# Wait for validation result
await validation_event.wait()
result = result_container["result"]
error = result_container["error"]
if error:
raise ValidationError(f"Validation error: {error}")
if result == ValidationResult.REJECT:
raise ValidationError("Message validation rejected")
elif result == ValidationResult.THROTTLED:
raise ValidationError("Message validation throttled")
elif result == ValidationResult.IGNORE:
# Treat IGNORE as rejection for now, or you could silently drop
raise ValidationError("Message validation ignored")
# ACCEPT case - just return normally
if not all(results):
raise ValidationError(f"Validation failed for msg={msg}")
async def push_msg(self, msg_forwarder: ID, msg: rpc_pb2.Message) -> None:
"""
@ -829,43 +778,3 @@ class Pubsub(Service, IPubsub):
def _is_subscribed_to_msg(self, msg: rpc_pb2.Message) -> bool:
return any(topic in self.topic_ids for topic in msg.topicIDs)
async def write_msg(self, stream: INetStream, rpc_msg: rpc_pb2.RPC) -> bool:
"""
Write an RPC message to a stream with proper error handling.
Implements WriteMsg similar to go-msgio which is used in go-libp2p
Ref: https://github.com/libp2p/go-msgio/blob/master/protoio/uvarint_writer.go#L56
:param stream: stream to write the message to
:param rpc_msg: RPC message to write
:return: True if successful, False if stream was closed
"""
try:
# Calculate message size first
msg_bytes = rpc_msg.SerializeToString()
msg_size = len(msg_bytes)
# Calculate varint size and allocate exact buffer size needed
varint_bytes = encode_uvarint(msg_size)
varint_size = len(varint_bytes)
# Allocate buffer with exact size (like Go's pool.Get())
buf = bytearray(varint_size + msg_size)
# Write varint length prefix to buffer (like Go's binary.PutUvarint())
buf[:varint_size] = varint_bytes
# Write serialized message after varint (like Go's rpc.MarshalTo())
buf[varint_size:] = msg_bytes
# Single write operation (like Go's s.Write(buf))
await stream.write(bytes(buf))
return True
except StreamClosed:
peer_id = stream.muxed_conn.peer_id
logger.debug("Fail to write message to %s: stream closed", peer_id)
self._handle_dead_peer(peer_id)
return False

View File

@ -1,314 +0,0 @@
from collections.abc import (
Callable,
)
from dataclasses import dataclass
from enum import Enum
import logging
from typing import (
NamedTuple,
cast,
)
import trio
from libp2p.custom_types import AsyncValidatorFn, ValidatorFn
from libp2p.peer.id import (
ID,
)
from .pb import (
rpc_pb2,
)
logger = logging.getLogger("libp2p.pubsub.validation")
class ValidationResult(Enum):
ACCEPT = "accept"
REJECT = "reject"
IGNORE = "ignore"
THROTTLED = "throttled"
@dataclass
class ValidationRequest:
"""Request for message validation"""
validators: list["TopicValidator"]
msg_forwarder: ID # peer ID
msg: rpc_pb2.Message # message object
result_callback: Callable[[ValidationResult, Exception | None], None]
class TopicValidator(NamedTuple):
topic: str
validator: ValidatorFn
is_async: bool
timeout: float | None = None
# Per-topic throttle semaphore
throttle_semaphore: trio.Semaphore | None = None
class ValidationThrottler:
"""Manages all validation throttling mechanisms"""
def __init__(
self,
queue_size: int = 32,
global_throttle_limit: int = 8192,
default_topic_throttle_limit: int = 1024,
worker_count: int | None = None,
):
# 1. Queue-level throttling - bounded memory channel
self._validation_send, self._validation_receive = trio.open_memory_channel[
ValidationRequest
](queue_size)
# 2. Global validation throttling - limits total concurrent async validations
self._global_throttle = trio.Semaphore(global_throttle_limit)
# 3. Per-topic throttling - each validator gets its own semaphore
self._default_topic_throttle_limit = default_topic_throttle_limit
# Worker management
# TODO: Find a better way to manage worker count
self._worker_count = worker_count or 4
self._running = False
async def start(self, nursery: trio.Nursery) -> None:
"""Start the validation workers"""
self._running = True
# Start validation worker tasks
for i in range(self._worker_count):
nursery.start_soon(self._validation_worker, f"worker-{i}")
async def stop(self) -> None:
"""Stop the validation system"""
self._running = False
await self._validation_send.aclose()
def create_topic_validator(
self,
topic: str,
validator: ValidatorFn,
is_async: bool,
timeout: float | None = None,
throttle_limit: int | None = None,
) -> TopicValidator:
"""Create a new topic validator with its own throttle"""
limit = throttle_limit or self._default_topic_throttle_limit
throttle_sem = trio.Semaphore(limit)
return TopicValidator(
topic=topic,
validator=validator,
is_async=is_async,
timeout=timeout,
throttle_semaphore=throttle_sem,
)
async def submit_validation(
self,
validators: list[TopicValidator],
msg_forwarder: ID,
msg: rpc_pb2.Message,
result_callback: Callable[[ValidationResult, Exception | None], None],
) -> bool:
"""
Submit a message for validation.
Returns True if queued successfully, False if queue is full (throttled).
"""
if not self._running:
result_callback(
ValidationResult.REJECT, Exception("Validation system not running")
)
return False
request = ValidationRequest(
validators=validators,
msg_forwarder=msg_forwarder,
msg=msg,
result_callback=result_callback,
)
try:
# This will raise trio.WouldBlock if queue is full
self._validation_send.send_nowait(request)
return True
except trio.WouldBlock:
# Queue-level throttling: drop the message
logger.debug(
"Validation queue full, dropping message from %s", msg_forwarder
)
result_callback(
ValidationResult.THROTTLED, Exception("Validation queue full")
)
return False
async def _validation_worker(self, worker_id: str) -> None:
"""Worker that processes validation requests"""
logger.debug("Validation worker %s started", worker_id)
async with self._validation_receive:
async for request in self._validation_receive:
if not self._running:
break
try:
# Process the validation request
result = await self._validate_message(request)
request.result_callback(result, None)
except Exception as e:
logger.exception("Error in validation worker %s", worker_id)
request.result_callback(ValidationResult.REJECT, e)
logger.debug("Validation worker %s stopped", worker_id)
async def _validate_message(self, request: ValidationRequest) -> ValidationResult:
"""Core validation logic with throttling"""
validators = request.validators
msg_forwarder = request.msg_forwarder
msg = request.msg
if not validators:
return ValidationResult.ACCEPT
# Separate sync and async validators
sync_validators = [v for v in validators if not v.is_async]
async_validators = [v for v in validators if v.is_async]
# Run synchronous validators first
for validator in sync_validators:
try:
# Apply per-topic throttling even for sync validators
if validator.throttle_semaphore:
validator.throttle_semaphore.acquire_nowait()
try:
result = validator.validator(msg_forwarder, msg)
if not result:
return ValidationResult.REJECT
finally:
validator.throttle_semaphore.release()
else:
result = validator.validator(msg_forwarder, msg)
if not result:
return ValidationResult.REJECT
except trio.WouldBlock:
# Per-topic throttling for sync validator
logger.debug("Sync validation throttled for topic %s", validator.topic)
return ValidationResult.THROTTLED
except Exception as e:
logger.exception(
"Sync validator failed for topic %s: %s", validator.topic, e
)
return ValidationResult.REJECT
# Handle async validators with global + per-topic throttling
if async_validators:
return await self._validate_async_validators(
async_validators, msg_forwarder, msg
)
return ValidationResult.ACCEPT
async def _validate_async_validators(
self, validators: list[TopicValidator], msg_forwarder: ID, msg: rpc_pb2.Message
) -> ValidationResult:
"""Handle async validators with proper throttling"""
if len(validators) == 1:
# Fast path for single validator
return await self._validate_single_async_validator(
validators[0], msg_forwarder, msg
)
# Multiple async validators - run them concurrently
try:
# Try to acquire global throttle slot
self._global_throttle.acquire_nowait()
except trio.WouldBlock:
logger.debug(
"Global validation throttle exceeded, dropping message from %s",
msg_forwarder,
)
return ValidationResult.THROTTLED
try:
async with trio.open_nursery() as nursery:
results = {}
async def run_validator(validator: TopicValidator, index: int) -> None:
"""Run a single async validator and store the result"""
nonlocal results
result = await self._validate_single_async_validator(
validator, msg_forwarder, msg
)
results[index] = result
# Start all validators concurrently
for i, validator in enumerate(validators):
nursery.start_soon(run_validator, validator, i)
# Process results - any reject or throttle causes overall failure
final_result = ValidationResult.ACCEPT
for result in results.values():
if result == ValidationResult.REJECT:
return ValidationResult.REJECT
elif result == ValidationResult.THROTTLED:
final_result = ValidationResult.THROTTLED
elif (
result == ValidationResult.IGNORE
and final_result == ValidationResult.ACCEPT
):
final_result = ValidationResult.IGNORE
return final_result
finally:
self._global_throttle.release()
return ValidationResult.IGNORE
async def _validate_single_async_validator(
self, validator: TopicValidator, msg_forwarder: ID, msg: rpc_pb2.Message
) -> ValidationResult:
"""Validate with a single async validator"""
# Apply per-topic throttling
if validator.throttle_semaphore:
try:
validator.throttle_semaphore.acquire_nowait()
except trio.WouldBlock:
logger.debug(
"Per-topic validation throttled for topic %s", validator.topic
)
return ValidationResult.THROTTLED
else:
# Fallback if no throttle semaphore configured
pass
try:
# Apply timeout if configured
result: bool
if validator.timeout:
with trio.fail_after(validator.timeout):
func = cast(AsyncValidatorFn, validator.validator)
result = await func(msg_forwarder, msg)
else:
func = cast(AsyncValidatorFn, validator.validator)
result = await func(msg_forwarder, msg)
return ValidationResult.ACCEPT if result else ValidationResult.REJECT
except trio.TooSlowError:
logger.debug("Validation timeout for topic %s", validator.topic)
return ValidationResult.IGNORE
except Exception as e:
logger.exception(
"Async validator failed for topic %s: %s", validator.topic, e
)
return ValidationResult.REJECT
finally:
if validator.throttle_semaphore:
validator.throttle_semaphore.release()
return ValidationResult.IGNORE

View File

@ -234,8 +234,7 @@ class RelayDiscovery(Service):
if not callable(proto_getter):
return None
if peer_id not in peerstore.peer_ids():
return None
try:
# Try to get protocols
proto_result = proto_getter(peer_id)
@ -284,6 +283,8 @@ class RelayDiscovery(Service):
return None
mux = self.host.get_mux()
if not hasattr(mux, "protocols"):
return None
peer_protocols = set()
# Get protocols from mux with proper type safety
@ -292,9 +293,7 @@ class RelayDiscovery(Service):
# Get protocols with proper typing
mux_protocols = mux.get_protocols()
if isinstance(mux_protocols, (list, tuple)):
available_protocols = [
p for p in mux.get_protocols() if p is not None
]
available_protocols = list(mux_protocols)
for protocol in available_protocols:
try:
@ -314,7 +313,7 @@ class RelayDiscovery(Service):
self._protocol_cache[peer_id] = peer_protocols
protocol_str = str(PROTOCOL_ID)
for protocol in map(TProtocol, peer_protocols):
for protocol in peer_protocols:
if protocol == protocol_str:
return True
return False

View File

@ -31,6 +31,9 @@ from libp2p.stream_muxer.yamux.yamux import (
Yamux,
)
# FIXME: add negotiate timeout to `MuxerMultistream`
DEFAULT_NEGOTIATE_TIMEOUT = 60
class MuxerMultistream:
"""

View File

@ -98,31 +98,15 @@ class YamuxStream(IMuxedStream):
# Flow control: Check if we have enough send window
total_len = len(data)
sent = 0
logging.debug(f"Stream {self.stream_id}: Starts writing {total_len} bytes ")
while sent < total_len:
# Wait for available window with timeout
timeout = False
async with self.window_lock:
if self.send_window == 0:
logging.debug(
f"Stream {self.stream_id}: Window is zero, waiting for update"
)
# Release lock and wait with timeout
self.window_lock.release()
# To avoid re-acquiring the lock immediately,
with trio.move_on_after(5.0) as cancel_scope:
while self.send_window == 0 and not self.closed:
await trio.sleep(0.01)
# If we timed out, cancel the scope
timeout = cancel_scope.cancelled_caught
# Re-acquire lock
await self.window_lock.acquire()
# If we timed out waiting for window update, raise an error
if timeout:
raise MuxedStreamError(
"Timed out waiting for window update after 5 seconds."
)
while sent < total_len:
async with self.window_lock:
# Wait for available window
while self.send_window == 0 and not self.closed:
# Release lock while waiting
self.window_lock.release()
await trio.sleep(0.01)
await self.window_lock.acquire()
if self.closed:
raise MuxedStreamError("Stream is closed")
@ -139,45 +123,25 @@ class YamuxStream(IMuxedStream):
await self.conn.secured_conn.write(header + chunk)
sent += to_send
async def send_window_update(self, increment: int, skip_lock: bool = False) -> None:
"""
Send a window update to peer.
# If window is getting low, consider updating
if self.send_window < DEFAULT_WINDOW_SIZE // 2:
await self.send_window_update()
async def send_window_update(self, increment: int | None = None) -> None:
"""Send a window update to peer."""
if increment is None:
increment = DEFAULT_WINDOW_SIZE - self.recv_window
param:increment: The amount to increment the window size by.
If None, uses the difference between DEFAULT_WINDOW_SIZE
and current receive window.
param:skip_lock (bool): If True, skips acquiring window_lock.
This should only be used when calling from a context
that already holds the lock.
"""
if increment <= 0:
# If increment is zero or negative, skip sending update
logging.debug(
f"Stream {self.stream_id}: Skipping window update"
f"(increment={increment})"
)
return
logging.debug(
f"Stream {self.stream_id}: Sending window update with increment={increment}"
)
async def _do_window_update() -> None:
async with self.window_lock:
self.recv_window += increment
header = struct.pack(
YAMUX_HEADER_FORMAT,
0,
TYPE_WINDOW_UPDATE,
0,
self.stream_id,
increment,
YAMUX_HEADER_FORMAT, 0, TYPE_WINDOW_UPDATE, 0, self.stream_id, increment
)
await self.conn.secured_conn.write(header)
if skip_lock:
await _do_window_update()
else:
async with self.window_lock:
await _do_window_update()
async def read(self, n: int | None = -1) -> bytes:
# Handle None value for n by converting it to -1
if n is None:
@ -190,68 +154,55 @@ class YamuxStream(IMuxedStream):
)
raise MuxedStreamEOF("Stream is closed for receiving")
# If reading until EOF (n == -1), block until stream is closed
if n == -1:
data = b""
while not self.conn.event_shutting_down.is_set():
while not self.recv_closed and not self.conn.event_shutting_down.is_set():
# Check if there's data in the buffer
buffer = self.conn.stream_buffers.get(self.stream_id)
if buffer and len(buffer) > 0:
# Wait for closure even if data is available
logging.debug(
f"Stream {self.stream_id}:Waiting for FIN before returning data"
)
await self.conn.stream_events[self.stream_id].wait()
self.conn.stream_events[self.stream_id] = trio.Event()
else:
# No data, wait for data or closure
logging.debug(f"Stream {self.stream_id}: Waiting for data or FIN")
await self.conn.stream_events[self.stream_id].wait()
self.conn.stream_events[self.stream_id] = trio.Event()
# If buffer is not available, check if stream is closed
# After loop, check if stream is closed or shutting down
async with self.conn.streams_lock:
if self.conn.event_shutting_down.is_set():
logging.debug(f"Stream {self.stream_id}: Connection shutting down")
raise MuxedStreamEOF("Connection shut down")
if self.closed:
if self.reset_received:
logging.debug(f"Stream {self.stream_id}: Stream was reset")
raise MuxedStreamReset("Stream was reset")
else:
logging.debug(
f"Stream {self.stream_id}: Stream closed cleanly (EOF)"
)
raise MuxedStreamEOF("Stream closed cleanly (EOF)")
buffer = self.conn.stream_buffers.get(self.stream_id)
if buffer is None:
logging.debug(f"Stream {self.stream_id}: No buffer available")
logging.debug(
f"Stream {self.stream_id}: Buffer gone, assuming closed"
)
raise MuxedStreamEOF("Stream buffer closed")
# If we have data in buffer, process it
if len(buffer) > 0:
chunk = bytes(buffer)
buffer.clear()
data += chunk
# Send window update for the chunk we just read
async with self.window_lock:
self.recv_window += len(chunk)
logging.debug(f"Stream {self.stream_id}: Update {len(chunk)}")
await self.send_window_update(len(chunk), skip_lock=True)
# If stream is closed (FIN received) and buffer is empty, break
if self.recv_closed and len(buffer) == 0:
logging.debug(f"Stream {self.stream_id}: Closed with empty buffer")
break
# If stream was reset, raise reset error
if self.reset_received:
logging.debug(f"Stream {self.stream_id}: Stream was reset")
raise MuxedStreamReset("Stream was reset")
# Wait for more data or stream closure
logging.debug(f"Stream {self.stream_id}: Waiting for data or FIN")
await self.conn.stream_events[self.stream_id].wait()
self.conn.stream_events[self.stream_id] = trio.Event()
# After loop exit, first check if we have data to return
if data:
logging.debug(
f"Stream {self.stream_id}: Returning {len(data)} bytes after loop"
)
logging.debug(f"Stream {self.stream_id}: EOF reached")
raise MuxedStreamEOF("Stream is closed for receiving")
# Return all buffered data
data = bytes(buffer)
buffer.clear()
logging.debug(f"Stream {self.stream_id}: Returning {len(data)} bytes")
return data
# No data accumulated, now check why we exited the loop
if self.conn.event_shutting_down.is_set():
logging.debug(f"Stream {self.stream_id}: Connection shutting down")
raise MuxedStreamEOF("Connection shut down")
# Return empty data
return b""
else:
data = await self.conn.read_stream(self.stream_id, n)
async with self.window_lock:
self.recv_window += len(data)
logging.debug(
f"Stream {self.stream_id}: Sending window update after read, "
f"increment={len(data)}"
)
await self.send_window_update(len(data), skip_lock=True)
return data
# For specific size read (n > 0), return available data immediately
return await self.conn.read_stream(self.stream_id, n)
async def close(self) -> None:
if not self.send_closed:
@ -542,7 +493,7 @@ class Yamux(IMuxedConn):
f"type={typ}, flags={flags}, stream_id={stream_id},"
f"length={length}"
)
if (typ == TYPE_DATA or typ == TYPE_WINDOW_UPDATE) and flags & FLAG_SYN:
if typ == TYPE_DATA and flags & FLAG_SYN:
async with self.streams_lock:
if stream_id not in self.streams:
stream = YamuxStream(stream_id, self, False)

View File

@ -7,8 +7,6 @@ from libp2p.utils.varint import (
encode_varint_prefixed,
read_delim,
read_varint_prefixed_bytes,
decode_varint_from_bytes,
decode_varint_with_size,
)
from libp2p.utils.version import (
get_agent_version,
@ -22,6 +20,4 @@ __all__ = [
"get_agent_version",
"read_delim",
"read_varint_prefixed_bytes",
"decode_varint_from_bytes",
"decode_varint_with_size",
]

View File

@ -39,38 +39,12 @@ def encode_uvarint(number: int) -> bytes:
return buf
def decode_varint_from_bytes(data: bytes) -> int:
"""
Decode a varint from bytes and return the value.
This is a synchronous version of decode_uvarint_from_stream for already-read bytes.
"""
res = 0
for shift in itertools.count(0, 7):
if shift > SHIFT_64_BIT_MAX:
raise ParseError("Integer is too large...")
if not data:
raise ParseError("Unexpected end of data")
value = data[0]
data = data[1:]
res += (value & LOW_MASK) << shift
if not value & HIGH_MASK:
break
return res
async def decode_uvarint_from_stream(reader: Reader) -> int:
"""https://en.wikipedia.org/wiki/LEB128."""
res = 0
for shift in itertools.count(0, 7):
if shift > SHIFT_64_BIT_MAX:
raise ParseError(
"Varint decoding error: integer exceeds maximum size of 64 bits."
)
raise ParseError("TODO: better exception msg: Integer is too large...")
byte = await read_exactly(reader, 1)
value = byte[0]
@ -82,33 +56,6 @@ async def decode_uvarint_from_stream(reader: Reader) -> int:
return res
def decode_varint_with_size(data: bytes) -> tuple[int, int]:
"""
Decode a varint from bytes and return (value, bytes_consumed).
Returns (0, 0) if the data doesn't start with a valid varint.
"""
try:
# Calculate how many bytes the varint consumes
varint_size = 0
for i, byte in enumerate(data):
varint_size += 1
if (byte & 0x80) == 0:
break
if varint_size == 0:
return 0, 0
# Extract just the varint bytes
varint_bytes = data[:varint_size]
# Decode the varint
value = decode_varint_from_bytes(varint_bytes)
return value, varint_size
except Exception:
return 0, 0
def encode_varint_prefixed(msg_bytes: bytes) -> bytes:
varint_len = encode_uvarint(len(msg_bytes))
return varint_len + msg_bytes

View File

@ -0,0 +1 @@
Added support for ``Kademlia DHT`` in py-libp2p.

View File

@ -0,0 +1,7 @@
Store public key and peer ID in peerstore during handshake
Modified the InsecureTransport class to accept an optional peerstore parameter and updated the handshake process to store the received public key and peer ID in the peerstore when available.
Added test cases to verify:
1. The peerstore remains unchanged when handshake fails due to peer ID mismatch
2. The handshake correctly adds a public key to a peer ID that already exists in the peerstore but doesn't have a public key yet

View File

@ -0,0 +1 @@
Refactored gossipsub heartbeat logic to use a single helper method `_handle_topic_heartbeat` that handles both fanout and gossip heartbeats.

View File

@ -0,0 +1 @@
Added sparse connect utility function to pubsub test utilities for creating test networks with configurable connectivity.

View File

@ -0,0 +1,2 @@
Reordered the arguments to `upgrade_security` to place `is_initiator` before `peer_id`, and made `peer_id` optional.
This allows the method to reflect the fact that peer identity is not required for inbound connections.

View File

@ -0,0 +1 @@
Uses the `decapsulate` method of the `Multiaddr` class to clean up the observed address.

View File

@ -0,0 +1 @@
Optimized pubsub publishing to send multiple topics in a single message instead of separate messages per topic.

View File

@ -0,0 +1 @@
added peer exchange and backoff logic as part of Gossipsub v1.1 upgrade

View File

@ -0,0 +1 @@
Fixed an issue in `Pubsub` where async validators were not handled reliably under concurrency. Now uses a safe aggregator list for consistent behavior.

View File

@ -1,3 +0,0 @@
Improved type safety in `get_mux()` and `get_protocols()` by returning properly typed values instead
of `Any`. Also updated `identify.py` and `discovery.py` to handle `None` values safely and
compare protocols correctly.

View File

@ -1 +0,0 @@
Add comprehensive tests for relay_discovery method in circuit_relay_v2

View File

@ -1 +0,0 @@
Add logic to clear_peerdata method in peerstore

View File

@ -1 +0,0 @@
fixed malformed PeerId in test_peerinfo

View File

@ -1 +0,0 @@
fixed a typecheck error using cast in peerinfo.py

View File

@ -1 +0,0 @@
Improve error message under the function decode_uvarint_from_stream in libp2p/utils/varint.py file

View File

@ -1 +0,0 @@
identify protocol use now prefix-length messages by default. use use_varint_format param for old raw messages

View File

@ -1 +0,0 @@
add length-prefixed support to identify protocol

View File

@ -1 +0,0 @@
Fix raw format reading in identify/push protocol and add comprehensive test coverage for both varint and raw formats

View File

@ -1,10 +1,11 @@
[build-system]
requires = ["setuptools>=42", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "libp2p"
version = "0.2.9"
version = "0.2.8"
description = "libp2p: The Python implementation of the libp2p networking stack"
readme = "README.md"
requires-python = ">=3.10, <4.0"
@ -22,7 +23,7 @@ dependencies = [
"multiaddr>=0.0.9",
"mypy-protobuf>=3.0.0",
"noiseprotocol>=0.3.0",
"protobuf>=4.21.0,<5.0.0",
"protobuf>=3.20.1,<4.0.0",
"pycryptodome>=3.9.2",
"pymultihash>=0.8.2",
"pynacl>=1.3.0",
@ -30,7 +31,6 @@ dependencies = [
"trio-typing>=0.0.4",
"trio>=0.26.0",
"fastecdsa==2.3.2; sys_platform != 'win32'",
"zeroconf (>=0.147.0,<0.148.0)",
]
classifiers = [
"Development Status :: 4 - Beta",
@ -54,7 +54,6 @@ identify-demo = "examples.identify.identify:main"
identify-push-demo = "examples.identify_push.identify_push_demo:run_main"
identify-push-listener-dialer-demo = "examples.identify_push.identify_push_listener_dialer:main"
pubsub-demo = "examples.pubsub.pubsub:main"
mdns-demo = "examples.mDNS.mDNS:main"
[project.optional-dependencies]
dev = [
@ -188,7 +187,7 @@ name = "Removals"
showcontent = true
[tool.bumpversion]
current_version = "0.2.9"
current_version = "0.2.8"
parse = """
(?P<major>\\d+)
\\.(?P<minor>\\d+)

View File

@ -11,7 +11,9 @@ from libp2p.identity.identify.identify import (
PROTOCOL_VERSION,
_mk_identify_protobuf,
_multiaddr_to_bytes,
parse_identify_response,
)
from libp2p.identity.identify.pb.identify_pb2 import (
Identify,
)
from tests.utils.factories import (
host_pair_factory,
@ -27,18 +29,14 @@ async def test_identify_protocol(security_protocol):
host_b,
):
# Here, host_b is the requester and host_a is the responder.
# observed_addr represent host_b's address as observed by host_a
# (i.e., the address from which host_b's request was received).
# observed_addr represent host_bs address as observed by host_a
# (i.e., the address from which host_bs request was received).
stream = await host_b.new_stream(host_a.get_id(), (ID,))
# Read the response (could be either format)
# Read a larger chunk to get all the data before stream closes
response = await stream.read(8192) # Read enough data in one go
response = await stream.read()
await stream.close()
# Parse the response (handles both old and new formats)
identify_response = parse_identify_response(response)
identify_response = Identify()
identify_response.ParseFromString(response)
logger.debug("host_a: %s", host_a.get_addrs())
logger.debug("host_b: %s", host_b.get_addrs())
@ -64,9 +62,8 @@ async def test_identify_protocol(security_protocol):
logger.debug("observed_addr: %s", Multiaddr(identify_response.observed_addr))
logger.debug("host_b.get_addrs()[0]: %s", host_b.get_addrs()[0])
# The observed address should match the cleaned address
assert Multiaddr(identify_response.observed_addr) == cleaned_addr
logger.debug("cleaned_addr= %s", cleaned_addr)
assert identify_response.observed_addr == _multiaddr_to_bytes(cleaned_addr)
# Check protocols
assert set(identify_response.protocols) == set(host_a.get_mux().get_protocols())

View File

@ -1,410 +0,0 @@
import pytest
from libp2p.identity.identify.identify import (
_mk_identify_protobuf,
)
from libp2p.identity.identify.pb.identify_pb2 import (
Identify,
)
from libp2p.io.abc import Closer, Reader, Writer
from libp2p.utils.varint import (
decode_varint_from_bytes,
encode_varint_prefixed,
)
from tests.utils.factories import (
host_pair_factory,
)
class MockStream(Reader, Writer, Closer):
"""Mock stream for testing identify protocol compatibility."""
def __init__(self, data: bytes):
self.data = data
self.position = 0
self.closed = False
async def read(self, n: int | None = None) -> bytes:
if self.closed or self.position >= len(self.data):
return b""
if n is None:
n = len(self.data) - self.position
result = self.data[self.position : self.position + n]
self.position += len(result)
return result
async def write(self, data: bytes) -> None:
# Mock write - just store the data
pass
async def close(self) -> None:
self.closed = True
def create_identify_message(host, observed_multiaddr=None):
"""Create an identify protobuf message."""
return _mk_identify_protobuf(host, observed_multiaddr)
def create_new_format_message(identify_msg):
"""Create a new format (length-prefixed) identify message."""
msg_bytes = identify_msg.SerializeToString()
return encode_varint_prefixed(msg_bytes)
def create_old_format_message(identify_msg):
"""Create an old format (raw protobuf) identify message."""
return identify_msg.SerializeToString()
async def read_new_format_message(stream) -> bytes:
"""Read a new format (length-prefixed) identify message."""
# Read varint length prefix
length_bytes = b""
while True:
b = await stream.read(1)
if not b:
break
length_bytes += b
if b[0] & 0x80 == 0:
break
if not length_bytes:
raise ValueError("No length prefix received")
msg_length = decode_varint_from_bytes(length_bytes)
# Read the protobuf message
response = await stream.read(msg_length)
if len(response) != msg_length:
raise ValueError("Incomplete message received")
return response
async def read_old_format_message(stream) -> bytes:
"""Read an old format (raw protobuf) identify message."""
# Read all available data
response = b""
while True:
chunk = await stream.read(4096)
if not chunk:
break
response += chunk
return response
async def read_compatible_message(stream) -> bytes:
"""Read an identify message in either old or new format."""
# Try to read a few bytes to detect the format
first_bytes = await stream.read(10)
if not first_bytes:
raise ValueError("No data received")
# Try to decode as varint length prefix (new format)
try:
msg_length = decode_varint_from_bytes(first_bytes)
# Validate that the length is reasonable (not too large)
if msg_length > 0 and msg_length <= 1024 * 1024: # Max 1MB
# Calculate how many bytes the varint consumed
varint_len = 0
for i, byte in enumerate(first_bytes):
varint_len += 1
if (byte & 0x80) == 0:
break
# Read the remaining protobuf message
remaining_bytes = await stream.read(
msg_length - (len(first_bytes) - varint_len)
)
if len(remaining_bytes) == msg_length - (len(first_bytes) - varint_len):
message_data = first_bytes[varint_len:] + remaining_bytes
# Try to parse as protobuf to validate
try:
Identify().ParseFromString(message_data)
return message_data
except Exception:
# If protobuf parsing fails, fall back to old format
pass
except Exception:
pass
# Fall back to old format (raw protobuf)
response = first_bytes
# Read more data if available
while True:
chunk = await stream.read(4096)
if not chunk:
break
response += chunk
return response
async def read_compatible_message_simple(stream) -> bytes:
"""Read a message in either old or new format (simplified version for testing)."""
# Try to read a few bytes to detect the format
first_bytes = await stream.read(10)
if not first_bytes:
raise ValueError("No data received")
# Try to decode as varint length prefix (new format)
try:
msg_length = decode_varint_from_bytes(first_bytes)
# Validate that the length is reasonable (not too large)
if msg_length > 0 and msg_length <= 1024 * 1024: # Max 1MB
# Calculate how many bytes the varint consumed
varint_len = 0
for i, byte in enumerate(first_bytes):
varint_len += 1
if (byte & 0x80) == 0:
break
# Read the remaining message
remaining_bytes = await stream.read(
msg_length - (len(first_bytes) - varint_len)
)
if len(remaining_bytes) == msg_length - (len(first_bytes) - varint_len):
return first_bytes[varint_len:] + remaining_bytes
except Exception:
pass
# Fall back to old format (raw data)
response = first_bytes
# Read more data if available
while True:
chunk = await stream.read(4096)
if not chunk:
break
response += chunk
return response
def detect_format(data):
"""Detect if data is in new or old format (varint-prefixed or raw protobuf)."""
if not data:
return "unknown"
# Try to decode as varint
try:
msg_length = decode_varint_from_bytes(data)
# Validate that the length is reasonable
if msg_length > 0 and msg_length <= 1024 * 1024: # Max 1MB
# Calculate varint length
varint_len = 0
for i, byte in enumerate(data):
varint_len += 1
if (byte & 0x80) == 0:
break
# Check if we have enough data for the message
if len(data) >= varint_len + msg_length:
# Additional check: try to parse the message as protobuf
try:
message_data = data[varint_len : varint_len + msg_length]
Identify().ParseFromString(message_data)
return "new"
except Exception:
# If protobuf parsing fails, it's probably not a valid new format
pass
except Exception:
pass
# If varint decoding fails or length is unreasonable, assume old format
return "old"
@pytest.mark.trio
async def test_identify_new_format_compatibility(security_protocol):
"""Test that identify protocol works with new format (length-prefixed) messages."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Create new format message
new_format_data = create_new_format_message(identify_msg)
# Create mock stream with new format data
stream = MockStream(new_format_data)
# Read using new format reader
response = await read_new_format_message(stream)
# Parse the response
parsed_msg = Identify()
parsed_msg.ParseFromString(response)
# Verify the message content
assert parsed_msg.protocol_version == identify_msg.protocol_version
assert parsed_msg.agent_version == identify_msg.agent_version
assert parsed_msg.public_key == identify_msg.public_key
@pytest.mark.trio
async def test_identify_old_format_compatibility(security_protocol):
"""Test that identify protocol works with old format (raw protobuf) messages."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Create old format message
old_format_data = create_old_format_message(identify_msg)
# Create mock stream with old format data
stream = MockStream(old_format_data)
# Read using old format reader
response = await read_old_format_message(stream)
# Parse the response
parsed_msg = Identify()
parsed_msg.ParseFromString(response)
# Verify the message content
assert parsed_msg.protocol_version == identify_msg.protocol_version
assert parsed_msg.agent_version == identify_msg.agent_version
assert parsed_msg.public_key == identify_msg.public_key
@pytest.mark.trio
async def test_identify_backward_compatibility_old_format(security_protocol):
"""Test backward compatibility reader with old format messages."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Create old format message
old_format_data = create_old_format_message(identify_msg)
# Create mock stream with old format data
stream = MockStream(old_format_data)
# Read using old format reader (which should work reliably)
response = await read_old_format_message(stream)
# Parse the response
parsed_msg = Identify()
parsed_msg.ParseFromString(response)
# Verify the message content
assert parsed_msg.protocol_version == identify_msg.protocol_version
assert parsed_msg.agent_version == identify_msg.agent_version
assert parsed_msg.public_key == identify_msg.public_key
@pytest.mark.trio
async def test_identify_backward_compatibility_new_format(security_protocol):
"""Test backward compatibility reader with new format messages."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Create new format message
new_format_data = create_new_format_message(identify_msg)
# Create mock stream with new format data
stream = MockStream(new_format_data)
# Read using new format reader (which should work reliably)
response = await read_new_format_message(stream)
# Parse the response
parsed_msg = Identify()
parsed_msg.ParseFromString(response)
# Verify the message content
assert parsed_msg.protocol_version == identify_msg.protocol_version
assert parsed_msg.agent_version == identify_msg.agent_version
assert parsed_msg.public_key == identify_msg.public_key
@pytest.mark.trio
async def test_identify_format_detection(security_protocol):
"""Test that the format detection works correctly."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Test new format detection
new_format_data = create_new_format_message(identify_msg)
format_type = detect_format(new_format_data)
assert format_type == "new", "New format should be detected correctly"
# Test old format detection
old_format_data = create_old_format_message(identify_msg)
format_type = detect_format(old_format_data)
assert format_type == "old", "Old format should be detected correctly"
@pytest.mark.trio
async def test_identify_error_handling(security_protocol):
"""Test error handling for malformed messages."""
from libp2p.exceptions import ParseError
# Test with empty data
stream = MockStream(b"")
with pytest.raises(ValueError, match="No data received"):
await read_compatible_message(stream)
# Test with incomplete varint
stream = MockStream(b"\x80") # Incomplete varint
with pytest.raises(ParseError, match="Unexpected end of data"):
await read_new_format_message(stream)
# Test with invalid protobuf data
stream = MockStream(b"\x05invalid") # Length prefix but invalid protobuf
with pytest.raises(Exception): # Should fail when parsing protobuf
response = await read_new_format_message(stream)
Identify().ParseFromString(response)
@pytest.mark.trio
async def test_identify_message_equivalence(security_protocol):
"""Test that old and new format messages are equivalent."""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Create identify message
identify_msg = create_identify_message(host_a)
# Create both formats
new_format_data = create_new_format_message(identify_msg)
old_format_data = create_old_format_message(identify_msg)
# Extract the protobuf message from new format
varint_len = 0
for i, byte in enumerate(new_format_data):
varint_len += 1
if (byte & 0x80) == 0:
break
new_format_protobuf = new_format_data[varint_len:]
# The protobuf messages should be identical
assert new_format_protobuf == old_format_data, (
"Protobuf messages should be identical in both formats"
)

View File

@ -1,7 +1,4 @@
import logging
from unittest.mock import (
patch,
)
import pytest
import multiaddr
@ -20,7 +17,6 @@ from libp2p.identity.identify.pb.identify_pb2 import (
Identify,
)
from libp2p.identity.identify_push.identify_push import (
CONCURRENCY_LIMIT,
ID_PUSH,
_update_peerstore_from_identify,
identify_push_handler_for,
@ -33,11 +29,6 @@ from libp2p.peer.peerinfo import (
from tests.utils.factories import (
host_pair_factory,
)
from tests.utils.utils import (
create_mock_connections,
run_host_forever,
wait_until_listening,
)
logger = logging.getLogger("libp2p.identity.identify-push-test")
@ -184,7 +175,6 @@ async def test_identify_push_to_peers(security_protocol):
host_c = new_host(key_pair=key_pair_c)
# Set up the identify/push handlers
host_a.set_stream_handler(ID_PUSH, identify_push_handler_for(host_a))
host_b.set_stream_handler(ID_PUSH, identify_push_handler_for(host_b))
host_c.set_stream_handler(ID_PUSH, identify_push_handler_for(host_c))
@ -214,20 +204,6 @@ async def test_identify_push_to_peers(security_protocol):
# Check that the peer is in the peerstore
assert peer_id_a in peerstore_c.peer_ids()
# Test for push_identify to only connected peers and not all peers
# Disconnect a from c.
await host_c.disconnect(host_a.get_id())
await push_identify_to_peers(host_c)
# Wait a bit for the push to complete
await trio.sleep(0.1)
# Check that host_a's peerstore has not been updated with host_c's info
assert host_c.get_id() not in host_a.get_peerstore().peer_ids()
# Check that host_b's peerstore has been updated with host_c's info
assert host_c.get_id() in host_b.get_peerstore().peer_ids()
@pytest.mark.trio
async def test_push_identify_to_peers_with_explicit_params(security_protocol):
@ -436,265 +412,3 @@ async def test_partial_update_peerstore_from_identify(security_protocol):
host_a_public_key = host_a.get_public_key().serialize()
peerstore_public_key = peerstore.pubkey(peer_id).serialize()
assert host_a_public_key == peerstore_public_key
@pytest.mark.trio
async def test_push_identify_to_peers_respects_concurrency_limit():
"""
Test bounded concurrency for the identify/push protocol to prevent
network congestion.
This test verifies:
1. The number of concurrent tasks executing the identify push is always
less than or equal to CONCURRENCY_LIMIT.
2. An error is raised if concurrency exceeds the defined limit.
It mocks `push_identify_to_peer` to simulate delay using sleep,
allowing the test to measure and assert actual concurrency behavior.
"""
state = {
"concurrency_counter": 0,
"max_observed": 0,
}
lock = trio.Lock()
async def mock_push_identify_to_peer(
host,
peer_id,
observed_multiaddr=None,
limit=trio.Semaphore(CONCURRENCY_LIMIT),
use_varint_format=True,
) -> bool:
"""
Mock function to test concurrency by simulating an identify message.
This function patches push_identify_to_peer for testing purpose
Returns
-------
bool
True if the push was successful, False otherwise.
"""
async with limit:
async with lock:
state["concurrency_counter"] += 1
if state["concurrency_counter"] > CONCURRENCY_LIMIT:
raise RuntimeError(
f"Concurrency limit exceeded: {state['concurrency_counter']}"
)
state["max_observed"] = max(
state["max_observed"], state["concurrency_counter"]
)
logger.debug("Successfully pushed identify to peer %s", peer_id)
await trio.sleep(0.05)
async with lock:
state["concurrency_counter"] -= 1
return True
# Create a mock host.
key_pair_host = create_new_key_pair()
host = new_host(key_pair=key_pair_host)
# Create a mock network and add mock connections to the host
host.get_network().connections = create_mock_connections()
with patch(
"libp2p.identity.identify_push.identify_push.push_identify_to_peer",
new=mock_push_identify_to_peer,
):
await push_identify_to_peers(host)
assert state["max_observed"] <= CONCURRENCY_LIMIT, (
f"Max concurrency observed: {state['max_observed']}"
)
@pytest.mark.trio
async def test_all_peers_receive_identify_push_with_semaphore(security_protocol):
dummy_peers = []
async with host_pair_factory(security_protocol=security_protocol) as (host_a, _):
# Create dummy peers
for _ in range(50):
key_pair = create_new_key_pair()
dummy_host = new_host(key_pair=key_pair)
dummy_host.set_stream_handler(
ID_PUSH, identify_push_handler_for(dummy_host)
)
listen_addr = multiaddr.Multiaddr("/ip4/127.0.0.1/tcp/0")
dummy_peers.append((dummy_host, listen_addr))
async with trio.open_nursery() as nursery:
# Start all dummy hosts
for host, listen_addr in dummy_peers:
nursery.start_soon(run_host_forever, host, listen_addr)
# Wait for all hosts to finish setting up listeners
for host, _ in dummy_peers:
await wait_until_listening(host)
# Now connect host_a → dummy peers
for host, _ in dummy_peers:
await host_a.connect(info_from_p2p_addr(host.get_addrs()[0]))
await push_identify_to_peers(
host_a,
)
await trio.sleep(0.5)
peer_id_a = host_a.get_id()
for host, _ in dummy_peers:
dummy_peerstore = host.get_peerstore()
assert peer_id_a in dummy_peerstore.peer_ids()
nursery.cancel_scope.cancel()
@pytest.mark.trio
async def test_all_peers_receive_identify_push_with_semaphore_under_high_peer_load(
security_protocol,
):
dummy_peers = []
async with host_pair_factory(security_protocol=security_protocol) as (host_a, _):
# Create dummy peers
# Breaking with more than 500 peers
# Trio have a async tasks limit of 1000
for _ in range(499):
key_pair = create_new_key_pair()
dummy_host = new_host(key_pair=key_pair)
dummy_host.set_stream_handler(
ID_PUSH, identify_push_handler_for(dummy_host)
)
listen_addr = multiaddr.Multiaddr("/ip4/127.0.0.1/tcp/0")
dummy_peers.append((dummy_host, listen_addr))
async with trio.open_nursery() as nursery:
# Start all dummy hosts
for host, listen_addr in dummy_peers:
nursery.start_soon(run_host_forever, host, listen_addr)
# Wait for all hosts to finish setting up listeners
for host, _ in dummy_peers:
await wait_until_listening(host)
# Now connect host_a → dummy peers
for host, _ in dummy_peers:
await host_a.connect(info_from_p2p_addr(host.get_addrs()[0]))
await push_identify_to_peers(
host_a,
)
await trio.sleep(0.5)
peer_id_a = host_a.get_id()
for host, _ in dummy_peers:
dummy_peerstore = host.get_peerstore()
assert peer_id_a in dummy_peerstore.peer_ids()
nursery.cancel_scope.cancel()
@pytest.mark.trio
async def test_identify_push_default_varint_format(security_protocol):
"""
Test that the identify/push protocol uses varint format by default.
This test verifies that:
1. The default behavior uses length-prefixed messages (varint format)
2. Messages are correctly encoded with varint length prefix
3. Messages are correctly decoded with varint length prefix
4. The peerstore is updated correctly with the received information
"""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Set up the identify/push handlers with default settings
# (use_varint_format=True)
host_b.set_stream_handler(ID_PUSH, identify_push_handler_for(host_b))
# Push identify information from host_a to host_b using default settings
success = await push_identify_to_peer(host_a, host_b.get_id())
assert success, "Identify push should succeed with default varint format"
# Wait a bit for the push to complete
await trio.sleep(0.1)
# Get the peerstore from host_b
peerstore = host_b.get_peerstore()
peer_id = host_a.get_id()
# Verify that the peerstore was updated correctly
assert peer_id in peerstore.peer_ids()
# Check that addresses have been updated
host_a_addrs = set(host_a.get_addrs())
peerstore_addrs = set(peerstore.addrs(peer_id))
assert all(addr in peerstore_addrs for addr in host_a_addrs)
# Check that protocols have been updated
host_a_protocols = set(host_a.get_mux().get_protocols())
peerstore_protocols = set(peerstore.get_protocols(peer_id))
assert all(protocol in peerstore_protocols for protocol in host_a_protocols)
# Check that the public key has been updated
host_a_public_key = host_a.get_public_key().serialize()
peerstore_public_key = peerstore.pubkey(peer_id).serialize()
assert host_a_public_key == peerstore_public_key
@pytest.mark.trio
async def test_identify_push_legacy_raw_format(security_protocol):
"""
Test that the identify/push protocol can use legacy raw format when specified.
This test verifies that:
1. When use_varint_format=False, messages are sent without length prefix
2. Raw protobuf messages are correctly encoded and decoded
3. The peerstore is updated correctly with the received information
4. The legacy format is backward compatible
"""
async with host_pair_factory(security_protocol=security_protocol) as (
host_a,
host_b,
):
# Set up the identify/push handlers with legacy format (use_varint_format=False)
host_b.set_stream_handler(
ID_PUSH, identify_push_handler_for(host_b, use_varint_format=False)
)
# Push identify information from host_a to host_b using legacy format
success = await push_identify_to_peer(
host_a, host_b.get_id(), use_varint_format=False
)
assert success, "Identify push should succeed with legacy raw format"
# Wait a bit for the push to complete
await trio.sleep(0.1)
# Get the peerstore from host_b
peerstore = host_b.get_peerstore()
peer_id = host_a.get_id()
# Verify that the peerstore was updated correctly
assert peer_id in peerstore.peer_ids()
# Check that addresses have been updated
host_a_addrs = set(host_a.get_addrs())
peerstore_addrs = set(peerstore.addrs(peer_id))
assert all(addr in peerstore_addrs for addr in host_a_addrs)
# Check that protocols have been updated
host_a_protocols = set(host_a.get_mux().get_protocols())
peerstore_protocols = set(peerstore.get_protocols(peer_id))
assert all(protocol in peerstore_protocols for protocol in host_a_protocols)
# Check that the public key has been updated
host_a_public_key = host_a.get_public_key().serialize()
peerstore_public_key = peerstore.pubkey(peer_id).serialize()
assert host_a_public_key == peerstore_public_key

View File

@ -6,12 +6,10 @@ from multiaddr import Multiaddr
from libp2p.crypto.secp256k1 import (
create_new_key_pair,
)
from libp2p.peer.id import ID
from libp2p.peer.peerdata import (
PeerData,
PeerDataError,
)
from libp2p.peer.peerstore import PeerStore
MOCK_ADDR = Multiaddr("/ip4/127.0.0.1/tcp/4001")
MOCK_KEYPAIR = create_new_key_pair()
@ -41,59 +39,6 @@ def test_set_protocols():
assert peer_data.get_protocols() == protocols
# Test case when removing protocols:
def test_remove_protocols():
peer_data = PeerData()
protocols: Sequence[str] = ["protocol1", "protocol2"]
peer_data.set_protocols(protocols)
peer_data.remove_protocols(["protocol1"])
assert peer_data.get_protocols() == ["protocol2"]
# Test case when clearing the protocol list:
def test_clear_protocol_data():
peer_data = PeerData()
protocols: Sequence[str] = ["protocol1", "protocol2"]
peer_data.set_protocols(protocols)
peer_data.clear_protocol_data()
assert peer_data.get_protocols() == []
# Test case when supports protocols:
def test_supports_protocols():
peer_data = PeerData()
peer_data.set_protocols(["protocol1", "protocol2", "protocol3"])
input_protocols = ["protocol1", "protocol4", "protocol2"]
supported = peer_data.supports_protocols(input_protocols)
assert supported == ["protocol1", "protocol2"]
# Test case for first supported protocol is found
def test_first_supported_protocol_found():
peer_data = PeerData()
peer_data.set_protocols(["protocolA", "protocolB"])
input_protocols = ["protocolC", "protocolB", "protocolA"]
first = peer_data.first_supported_protocol(input_protocols)
assert first == "protocolB"
# Test case for first supported protocol not found
def test_first_supported_protocol_none():
peer_data = PeerData()
peer_data.set_protocols(["protocolX", "protocolY"])
input_protocols = ["protocolA", "protocolB"]
first = peer_data.first_supported_protocol(input_protocols)
assert first == "None supported"
# Test case when adding addresses
def test_add_addrs():
peer_data = PeerData()
@ -136,15 +81,6 @@ def test_get_metadata_key_not_found():
peer_data.get_metadata("nonexistent_key")
# Test case for clearing metadata
def test_clear_metadata():
peer_data = PeerData()
peer_data.metadata = {"key1": "value1", "key2": "value2"}
peer_data.clear_metadata()
assert peer_data.metadata == {}
# Test case for adding public key
def test_add_pubkey():
peer_data = PeerData()
@ -171,71 +107,3 @@ def test_get_privkey_not_found():
peer_data = PeerData()
with pytest.raises(PeerDataError):
peer_data.get_privkey()
# Test case for returning all the peers with stored keys
def test_peer_with_keys():
peer_store = PeerStore()
peer_id_1 = ID(b"peer1")
peer_id_2 = ID(b"peer2")
peer_data_1 = PeerData()
peer_data_2 = PeerData()
peer_data_1.pubkey = MOCK_PUBKEY
peer_data_2.pubkey = None
peer_store.peer_data_map = {
peer_id_1: peer_data_1,
peer_id_2: peer_data_2,
}
assert peer_store.peer_with_keys() == [peer_id_1]
# Test case for clearing the key book
def test_clear_keydata():
peer_store = PeerStore()
peer_id = ID(b"peer123")
peer_data = PeerData()
peer_data.pubkey = MOCK_PUBKEY
peer_data.privkey = MOCK_PRIVKEY
peer_store.peer_data_map = {peer_id: peer_data}
peer_store.clear_keydata(peer_id)
assert peer_data.pubkey is None
assert peer_data.privkey is None
# Test case for recording latency for the first time
def test_record_latency_initial():
peer_data = PeerData()
assert peer_data.latency_EWMA() == 0
peer_data.record_latency(100.0)
assert peer_data.latency_EWMA() == 100.0
# Test case for updating latency
def test_record_latency_updates_ewma():
peer_data = PeerData()
peer_data.record_latency(100.0) # first measurement
first = peer_data.latency_EWMA()
peer_data.record_latency(50.0) # second measurement
second = peer_data.latency_EWMA()
assert second < first # EWMA should have smoothed downward
assert second > 50.0 # Not as low as the new latency
assert second != first
def test_clear_metrics():
peer_data = PeerData()
peer_data.record_latency(200.0)
assert peer_data.latency_EWMA() == 200.0
peer_data.clear_metrics()
assert peer_data.latency_EWMA() == 0

View File

@ -13,9 +13,7 @@ from libp2p.peer.peerinfo import (
)
ALPHABETS = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
VALID_MULTI_ADDR_STR = (
"/ip4/127.0.0.1/tcp/8000/p2p/QmWQqHcMi6Cay5M6KWSNVYSDnxzfqWb1aGFQFSRzBNe49t"
)
VALID_MULTI_ADDR_STR = "/ip4/127.0.0.1/tcp/8000/p2p/3YgLAeMKSAPcGqZkAt8mREqhQXmJT8SN8VCMN4T6ih4GNX9wvK8mWJnWZ1qA2mLdCQ" # noqa: E501
def test_init_():
@ -52,6 +50,9 @@ def test_info_from_p2p_addr_invalid(addr):
def test_info_from_p2p_addr_valid():
m_addr = multiaddr.Multiaddr(VALID_MULTI_ADDR_STR)
info = info_from_p2p_addr(m_addr)
assert info.peer_id.pretty() == "QmWQqHcMi6Cay5M6KWSNVYSDnxzfqWb1aGFQFSRzBNe49t"
assert (
info.peer_id.pretty()
== "3YgLAeMKSAPcGqZkAt8mREqhQXmJT8SN8VCMN4T6ih4GNX9wvK8mWJnWZ1qA2mLdCQ"
)
assert len(info.addrs) == 1
assert str(info.addrs[0]) == "/ip4/127.0.0.1/tcp/8000"

View File

@ -2,7 +2,6 @@ import time
import pytest
from multiaddr import Multiaddr
import trio
from libp2p.peer.id import ID
from libp2p.peer.peerstore import (
@ -90,33 +89,3 @@ def test_peers():
store.add_addr(ID(b"peer3"), Multiaddr("/ip4/127.0.0.1/tcp/4001"), 10)
assert set(store.peer_ids()) == {ID(b"peer1"), ID(b"peer2"), ID(b"peer3")}
@pytest.mark.trio
async def test_addr_stream_yields_new_addrs():
store = PeerStore()
peer_id = ID(b"peer1")
addr1 = Multiaddr("/ip4/127.0.0.1/tcp/4001")
addr2 = Multiaddr("/ip4/127.0.0.1/tcp/4002")
collected = []
async def consume_addrs():
async for addr in store.addr_stream(peer_id):
collected.append(addr)
if len(collected) == 2:
break
async with trio.open_nursery() as nursery:
nursery.start_soon(consume_addrs)
await trio.sleep(2) # Give time for the stream to start
store.add_addr(peer_id, addr1, ttl=10)
await trio.sleep(0.2)
store.add_addr(peer_id, addr2, ttl=10)
await trio.sleep(0.2)
# After collecting expected addresses, cancel the stream
nursery.cancel_scope.cancel()
assert collected == [addr1, addr2]

View File

@ -1,59 +0,0 @@
import pytest
import trio
from libp2p.abc import (
IMultiselectCommunicator,
)
from libp2p.custom_types import TProtocol
from libp2p.protocol_muxer.exceptions import (
MultiselectClientError,
MultiselectError,
)
from libp2p.protocol_muxer.multiselect import Multiselect
from libp2p.protocol_muxer.multiselect_client import MultiselectClient
class DummyMultiselectCommunicator(IMultiselectCommunicator):
"""
Dummy MultiSelectCommunicator to test out negotiate timmeout.
"""
def __init__(self) -> None:
return
async def write(self, msg_str: str) -> None:
"""Goes into infinite loop when .write is called"""
await trio.sleep_forever()
async def read(self) -> str:
"""Returns a dummy read"""
return "dummy_read"
@pytest.mark.trio
async def test_select_one_of_timeout():
ECHO = TProtocol("/echo/1.0.0")
communicator = DummyMultiselectCommunicator()
client = MultiselectClient()
with pytest.raises(MultiselectClientError, match="response timed out"):
await client.select_one_of([ECHO], communicator, 2)
@pytest.mark.trio
async def test_query_multistream_command_timeout():
communicator = DummyMultiselectCommunicator()
client = MultiselectClient()
with pytest.raises(MultiselectClientError, match="response timed out"):
await client.query_multistream_command(communicator, "ls", 2)
@pytest.mark.trio
async def test_negotiate_timeout():
communicator = DummyMultiselectCommunicator()
server = Multiselect()
with pytest.raises(MultiselectError, match="handshake read timeout"):
await server.negotiate(communicator, 2)

View File

@ -3,7 +3,6 @@ import pytest
from libp2p.custom_types import (
TProtocol,
)
from libp2p.protocol_muxer.multiselect import Multiselect
from libp2p.tools.utils import (
create_echo_stream_handler,
)
@ -139,23 +138,3 @@ async def test_multistream_command(security_protocol):
# Dialer asks for unspoorted command
with pytest.raises(ValueError, match="Command not supported"):
await dialer.send_command(listener.get_id(), "random")
@pytest.mark.trio
async def test_get_protocols_returns_all_registered_protocols():
ms = Multiselect()
async def dummy_handler(stream):
pass
p1 = TProtocol("/echo/1.0.0")
p2 = TProtocol("/foo/1.0.0")
p3 = TProtocol("/bar/1.0.0")
ms.add_handler(p1, dummy_handler)
ms.add_handler(p2, dummy_handler)
ms.add_handler(p3, dummy_handler)
protocols = ms.get_protocols()
assert set(protocols) == {p1, p2, p3}

View File

@ -15,7 +15,6 @@ from tests.utils.factories import (
PubsubFactory,
)
from tests.utils.pubsub.utils import (
connect_some,
dense_connect,
one_to_all_connect,
sparse_connect,
@ -591,166 +590,3 @@ async def test_sparse_connect():
f"received the message. Ideally all nodes should receive it, but at "
f"minimum {min_required} required for sparse network scalability."
)
@pytest.mark.trio
async def test_connect_some_with_fewer_hosts_than_degree():
"""Test connect_some when there are fewer hosts than degree."""
# Create 3 hosts with degree=5
async with PubsubFactory.create_batch_with_floodsub(3) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
degree = 5
await connect_some(hosts, degree)
await trio.sleep(0.1) # Allow connections to establish
# Each host should connect to all other hosts (since there are only 2 others)
for i, pubsub in enumerate(pubsubs_fsub):
connected_peers = len(pubsub.peers)
expected_max_connections = len(hosts) - 1 # All others
assert connected_peers <= expected_max_connections, (
f"Host {i} has {connected_peers} connections, "
f"but can only connect to {expected_max_connections} others"
)
@pytest.mark.trio
async def test_connect_some_degree_limit_enforced():
"""Test that connect_some enforces degree limits and creates expected topology."""
# Test with small network where we can verify exact behavior
async with PubsubFactory.create_batch_with_floodsub(6) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
degree = 2
await connect_some(hosts, degree)
await trio.sleep(0.1)
# With 6 hosts and degree=2, expected connections:
# Host 0 → connects to hosts 1,2 (2 peers total)
# Host 1 → connects to hosts 2,3 (3 peers: 0,2,3)
# Host 2 → connects to hosts 3,4 (4 peers: 0,1,3,4)
# Host 3 → connects to hosts 4,5 (3 peers: 1,2,4,5) - wait, that's 4!
# Host 4 → connects to host 5 (3 peers: 2,3,5)
# Host 5 → (2 peers: 3,4)
peer_counts = [len(pubsub.peers) for pubsub in pubsubs_fsub]
# First and last hosts should have exactly degree connections
assert peer_counts[0] == degree, (
f"Host 0 should have {degree} peers, got {peer_counts[0]}"
)
assert peer_counts[-1] <= degree, (
f"Last host should have ≤ {degree} peers, got {peer_counts[-1]}"
)
# Middle hosts may have more due to bidirectional connections
# but the pattern should be consistent with degree limit
total_connections = sum(peer_counts)
# Should be less than full mesh (each host connected to all others)
full_mesh_connections = len(hosts) * (len(hosts) - 1)
assert total_connections < full_mesh_connections, (
f"Got {total_connections} total connections, "
f"but full mesh would be {full_mesh_connections}"
)
# Should be more than just a chain (each host connected to next only)
chain_connections = 2 * (len(hosts) - 1) # bidirectional chain
assert total_connections > chain_connections, (
f"Got {total_connections} total connections, which is too few "
f"(chain would be {chain_connections})"
)
@pytest.mark.trio
async def test_connect_some_degree_zero():
"""Test edge case: degree=0 should result in no connections."""
# Create 5 hosts with degree=0
async with PubsubFactory.create_batch_with_floodsub(5) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
degree = 0
await connect_some(hosts, degree)
await trio.sleep(0.1) # Allow any potential connections to establish
# Verify no connections were made
for i, pubsub in enumerate(pubsubs_fsub):
connected_peers = len(pubsub.peers)
assert connected_peers == 0, (
f"Host {i} has {connected_peers} connections, "
f"but degree=0 should result in no connections"
)
@pytest.mark.trio
async def test_connect_some_negative_degree():
"""Test edge case: negative degree should be handled gracefully."""
# Create 5 hosts with degree=-1
async with PubsubFactory.create_batch_with_floodsub(5) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
degree = -1
await connect_some(hosts, degree)
await trio.sleep(0.1) # Allow any potential connections to establish
# Verify no connections were made (negative degree should behave like 0)
for i, pubsub in enumerate(pubsubs_fsub):
connected_peers = len(pubsub.peers)
assert connected_peers == 0, (
f"Host {i} has {connected_peers} connections, "
f"but negative degree should result in no connections"
)
@pytest.mark.trio
async def test_sparse_connect_degree_zero():
"""Test sparse_connect with degree=0."""
async with PubsubFactory.create_batch_with_floodsub(8) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
degree = 0
await sparse_connect(hosts, degree)
await trio.sleep(0.1) # Allow connections to establish
# With degree=0, sparse_connect should still create neighbor connections
# for connectivity (this is part of the algorithm design)
for i, pubsub in enumerate(pubsubs_fsub):
connected_peers = len(pubsub.peers)
# Should have some connections due to neighbor connectivity
# (each node connects to immediate neighbors)
expected_neighbors = 2 # previous and next in ring
assert connected_peers >= expected_neighbors, (
f"Host {i} has {connected_peers} connections, "
f"expected at least {expected_neighbors} neighbor connections"
)
@pytest.mark.trio
async def test_empty_host_list():
"""Test edge case: empty host list should be handled gracefully."""
hosts = []
# All functions should handle empty lists gracefully
await connect_some(hosts, 5)
await sparse_connect(hosts, 3)
await dense_connect(hosts)
# If we reach here without exceptions, the test passes
@pytest.mark.trio
async def test_single_host():
"""Test edge case: single host should be handled gracefully."""
async with PubsubFactory.create_batch_with_floodsub(1) as pubsubs_fsub:
hosts = [pubsub.host for pubsub in pubsubs_fsub]
# All functions should handle single host gracefully
await connect_some(hosts, 5)
await sparse_connect(hosts, 3)
await dense_connect(hosts)
# Single host should have no connections
connected_peers = len(pubsubs_fsub[0].peers)
assert connected_peers == 0, (
f"Single host has {connected_peers} connections, expected 0"
)

View File

@ -105,11 +105,11 @@ async def test_relay_discovery_initialization():
@pytest.mark.trio
async def test_relay_discovery_find_relay_peerstore_method():
"""Test finding a relay node via discovery using the peerstore method."""
async def test_relay_discovery_find_relay():
"""Test finding a relay node via discovery."""
async with HostFactory.create_batch_and_listen(2) as hosts:
relay_host, client_host = hosts
logger.info("Created host for test_relay_discovery_find_relay_peerstore_method")
logger.info("Created hosts for test_relay_discovery_find_relay")
logger.info("Relay host ID: %s", relay_host.get_id())
logger.info("Client host ID: %s", client_host.get_id())
@ -144,19 +144,19 @@ async def test_relay_discovery_find_relay_peerstore_method():
# Start discovery service
async with background_trio_service(client_discovery):
await client_discovery.event_started.wait()
logger.info("Client discovery service started (peerstore method)")
logger.info("Client discovery service started")
# Wait for discovery to find the relay using the peerstore method
logger.info("Waiting for relay discovery using peerstore...")
# Wait for discovery to find the relay
logger.info("Waiting for relay discovery...")
# Manually trigger discovery which uses peerstore as default
# Manually trigger discovery instead of waiting
await client_discovery.discover_relays()
# Check if relay was found
with trio.fail_after(DISCOVERY_TIMEOUT):
for _ in range(20): # Try multiple times
if relay_host.get_id() in client_discovery._discovered_relays:
logger.info("Relay discovered successfully (peerstore method)")
logger.info("Relay discovered successfully")
break
# Wait and try again
@ -164,194 +164,14 @@ async def test_relay_discovery_find_relay_peerstore_method():
# Manually trigger discovery again
await client_discovery.discover_relays()
else:
pytest.fail(
"Failed to discover relay node within timeout(peerstore method)"
)
pytest.fail("Failed to discover relay node within timeout")
# Verify that relay was found and is valid
assert relay_host.get_id() in client_discovery._discovered_relays, (
"Relay should be discovered (peerstore method)"
"Relay should be discovered"
)
relay_info = client_discovery._discovered_relays[relay_host.get_id()]
assert relay_info.peer_id == relay_host.get_id(), (
"Peer ID should match (peerstore method)"
)
@pytest.mark.trio
async def test_relay_discovery_find_relay_direct_connection_method():
"""Test finding a relay node via discovery using the direct connection method."""
async with HostFactory.create_batch_and_listen(2) as hosts:
relay_host, client_host = hosts
logger.info("Created hosts for test_relay_discovery_find_relay_direct_method")
logger.info("Relay host ID: %s", relay_host.get_id())
logger.info("Client host ID: %s", client_host.get_id())
# Explicitly register the protocol handlers on relay_host
relay_host.set_stream_handler(PROTOCOL_ID, simple_stream_handler)
relay_host.set_stream_handler(STOP_PROTOCOL_ID, simple_stream_handler)
# Manually add protocol to peerstore for testing, then remove to force fallback
client_host.get_peerstore().add_protocols(
relay_host.get_id(), [str(PROTOCOL_ID)]
)
# Set up discovery on the client host
client_discovery = RelayDiscovery(
client_host, discovery_interval=5
) # Use shorter interval for testing
try:
# Connect peers so they can discover each other
with trio.fail_after(CONNECT_TIMEOUT):
logger.info("Connecting client host to relay host")
await connect(client_host, relay_host)
assert relay_host.get_network().connections[client_host.get_id()], (
"Peers not connected"
)
logger.info("Connection established between peers")
except Exception as e:
logger.error("Failed to connect peers: %s", str(e))
raise
# Remove the relay from the peerstore to test fallback to direct connection
client_host.get_peerstore().clear_peerdata(relay_host.get_id())
# Make sure that peer_id is not present in peerstore
assert relay_host.get_id() not in client_host.get_peerstore().peer_ids()
# Start discovery service
async with background_trio_service(client_discovery):
await client_discovery.event_started.wait()
logger.info("Client discovery service started (direct connection method)")
# Wait for discovery to find the relay using the direct connection method
logger.info(
"Waiting for relay discovery using direct connection fallback..."
)
# Manually trigger discovery which should fallback to direct connection
await client_discovery.discover_relays()
# Check if relay was found
with trio.fail_after(DISCOVERY_TIMEOUT):
for _ in range(20): # Try multiple times
if relay_host.get_id() in client_discovery._discovered_relays:
logger.info("Relay discovered successfully (direct method)")
break
# Wait and try again
await trio.sleep(1)
# Manually trigger discovery again
await client_discovery.discover_relays()
else:
pytest.fail(
"Failed to discover relay node within timeout (direct method)"
)
# Verify that relay was found and is valid
assert relay_host.get_id() in client_discovery._discovered_relays, (
"Relay should be discovered (direct method)"
)
relay_info = client_discovery._discovered_relays[relay_host.get_id()]
assert relay_info.peer_id == relay_host.get_id(), (
"Peer ID should match (direct method)"
)
@pytest.mark.trio
async def test_relay_discovery_find_relay_mux_method():
"""
Test finding a relay node via discovery using the mux method
(fallback after direct connection fails).
"""
async with HostFactory.create_batch_and_listen(2) as hosts:
relay_host, client_host = hosts
logger.info("Created hosts for test_relay_discovery_find_relay_mux_method")
logger.info("Relay host ID: %s", relay_host.get_id())
logger.info("Client host ID: %s", client_host.get_id())
# Explicitly register the protocol handlers on relay_host
relay_host.set_stream_handler(PROTOCOL_ID, simple_stream_handler)
relay_host.set_stream_handler(STOP_PROTOCOL_ID, simple_stream_handler)
client_host.set_stream_handler(PROTOCOL_ID, simple_stream_handler)
client_host.set_stream_handler(STOP_PROTOCOL_ID, simple_stream_handler)
# Set up discovery on the client host
client_discovery = RelayDiscovery(
client_host, discovery_interval=5
) # Use shorter interval for testing
try:
# Connect peers so they can discover each other
with trio.fail_after(CONNECT_TIMEOUT):
logger.info("Connecting client host to relay host")
await connect(client_host, relay_host)
assert relay_host.get_network().connections[client_host.get_id()], (
"Peers not connected"
)
logger.info("Connection established between peers")
except Exception as e:
logger.error("Failed to connect peers: %s", str(e))
raise
# Remove the relay from the peerstore to test fallback
client_host.get_peerstore().clear_peerdata(relay_host.get_id())
# Make sure that peer_id is not present in peerstore
assert relay_host.get_id() not in client_host.get_peerstore().peer_ids()
# Mock the _check_via_direct_connection method to return None
# This forces the discovery to fall back to the mux method
async def mock_direct_check_fails(peer_id):
"""Mock that always returns None to force mux fallback."""
return None
client_discovery._check_via_direct_connection = mock_direct_check_fails
# Start discovery service
async with background_trio_service(client_discovery):
await client_discovery.event_started.wait()
logger.info("Client discovery service started (mux method)")
# Wait for discovery to find the relay using the mux method
logger.info("Waiting for relay discovery using mux fallback...")
# Manually trigger discovery which should fallback to mux method
await client_discovery.discover_relays()
# Check if relay was found
with trio.fail_after(DISCOVERY_TIMEOUT):
for _ in range(20): # Try multiple times
if relay_host.get_id() in client_discovery._discovered_relays:
logger.info("Relay discovered successfully (mux method)")
break
# Wait and try again
await trio.sleep(1)
# Manually trigger discovery again
await client_discovery.discover_relays()
else:
pytest.fail(
"Failed to discover relay node within timeout (mux method)"
)
# Verify that relay was found and is valid
assert relay_host.get_id() in client_discovery._discovered_relays, (
"Relay should be discovered (mux method)"
)
relay_info = client_discovery._discovered_relays[relay_host.get_id()]
assert relay_info.peer_id == relay_host.get_id(), (
"Peer ID should match (mux method)"
)
# Verify that the protocol was cached via mux method
assert relay_host.get_id() in client_discovery._protocol_cache, (
"Protocol should be cached (mux method)"
)
assert (
str(PROTOCOL_ID)
in client_discovery._protocol_cache[relay_host.get_id()]
), "Relay protocol should be in cache (mux method)"
assert relay_info.peer_id == relay_host.get_id(), "Peer ID should match"
@pytest.mark.trio

View File

@ -1,199 +0,0 @@
import logging
import pytest
import trio
from trio.testing import (
memory_stream_pair,
)
from libp2p.abc import IRawConnection
from libp2p.crypto.ed25519 import (
create_new_key_pair,
)
from libp2p.peer.id import (
ID,
)
from libp2p.security.insecure.transport import (
InsecureTransport,
)
from libp2p.stream_muxer.yamux.yamux import (
Yamux,
YamuxStream,
)
class TrioStreamAdapter(IRawConnection):
"""Adapter to make trio memory streams work with libp2p."""
def __init__(self, send_stream, receive_stream, is_initiator=False):
self.send_stream = send_stream
self.receive_stream = receive_stream
self.is_initiator = is_initiator
async def write(self, data: bytes) -> None:
logging.debug(f"Attempting to write {len(data)} bytes")
with trio.move_on_after(2):
await self.send_stream.send_all(data)
async def read(self, n: int | None = None) -> bytes:
if n is None or n <= 0:
raise ValueError("Reading unbounded or zero bytes not supported")
logging.debug(f"Attempting to read {n} bytes")
with trio.move_on_after(2):
data = await self.receive_stream.receive_some(n)
logging.debug(f"Read {len(data)} bytes")
return data
async def close(self) -> None:
logging.debug("Closing stream")
await self.send_stream.aclose()
await self.receive_stream.aclose()
def get_remote_address(self) -> tuple[str, int] | None:
"""Return None since this is a test adapter without real network info."""
return None
@pytest.fixture
def key_pair():
return create_new_key_pair()
@pytest.fixture
def peer_id(key_pair):
return ID.from_pubkey(key_pair.public_key)
@pytest.fixture
async def secure_conn_pair(key_pair, peer_id):
"""Create a pair of secure connections for testing."""
logging.debug("Setting up secure_conn_pair")
client_send, server_receive = memory_stream_pair()
server_send, client_receive = memory_stream_pair()
client_rw = TrioStreamAdapter(client_send, client_receive)
server_rw = TrioStreamAdapter(server_send, server_receive)
insecure_transport = InsecureTransport(key_pair)
async def run_outbound(nursery_results):
with trio.move_on_after(5):
client_conn = await insecure_transport.secure_outbound(client_rw, peer_id)
logging.debug("Outbound handshake complete")
nursery_results["client"] = client_conn
async def run_inbound(nursery_results):
with trio.move_on_after(5):
server_conn = await insecure_transport.secure_inbound(server_rw)
logging.debug("Inbound handshake complete")
nursery_results["server"] = server_conn
nursery_results = {}
async with trio.open_nursery() as nursery:
nursery.start_soon(run_outbound, nursery_results)
nursery.start_soon(run_inbound, nursery_results)
await trio.sleep(0.1) # Give tasks a chance to finish
client_conn = nursery_results.get("client")
server_conn = nursery_results.get("server")
if client_conn is None or server_conn is None:
raise RuntimeError("Handshake failed: client_conn or server_conn is None")
logging.debug("secure_conn_pair setup complete")
return client_conn, server_conn
@pytest.fixture
async def yamux_pair(secure_conn_pair, peer_id):
"""Create a pair of Yamux multiplexers for testing."""
logging.debug("Setting up yamux_pair")
client_conn, server_conn = secure_conn_pair
client_yamux = Yamux(client_conn, peer_id, is_initiator=True)
server_yamux = Yamux(server_conn, peer_id, is_initiator=False)
async with trio.open_nursery() as nursery:
with trio.move_on_after(5):
nursery.start_soon(client_yamux.start)
nursery.start_soon(server_yamux.start)
await trio.sleep(0.1)
logging.debug("yamux_pair started")
yield client_yamux, server_yamux
logging.debug("yamux_pair cleanup")
@pytest.mark.trio
async def test_yamux_race_condition_without_locks(yamux_pair):
"""
Test for race-around/interleaving in Yamux streams,when reading in
segments of data.
This launches concurrent writers/readers on both sides of a stream.
If there is no proper locking, the received data may be interleaved
or corrupted.
The test creates structured messages and verifies they are received
intact and in order.
Without proper locking, concurrent read/write operations could cause
data corruption
or message interleaving, which this test will catch.
"""
client_yamux, server_yamux = yamux_pair
client_stream: YamuxStream = await client_yamux.open_stream()
server_stream: YamuxStream = await server_yamux.accept_stream()
MSG_COUNT = 10
MSG_SIZE = 256 * 1024 # At max,only DEFAULT_WINDOW_SIZE bytes can be read
client_msgs = [
f"CLIENT-MSG-{i:03d}-".encode().ljust(MSG_SIZE, b"C") for i in range(MSG_COUNT)
]
server_msgs = [
f"SERVER-MSG-{i:03d}-".encode().ljust(MSG_SIZE, b"S") for i in range(MSG_COUNT)
]
client_received = []
server_received = []
async def writer(stream, msgs, name):
"""Write messages with minimal delays to encourage race conditions."""
for i, msg in enumerate(msgs):
await stream.write(msg)
# Yield control frequently to encourage interleaving
if i % 5 == 0:
await trio.sleep(0.005)
async def reader(stream, received, name):
"""Read messages and store them for verification."""
for i in range(MSG_COUNT):
data = await stream.read(MSG_SIZE)
received.append(data)
if i % 3 == 0:
await trio.sleep(0.001)
# Running all operations concurrently
async with trio.open_nursery() as nursery:
nursery.start_soon(writer, client_stream, client_msgs, "client")
nursery.start_soon(writer, server_stream, server_msgs, "server")
nursery.start_soon(reader, client_stream, client_received, "client")
nursery.start_soon(reader, server_stream, server_received, "server")
assert len(client_received) == MSG_COUNT, (
f"Client received {len(client_received)} messages, expected {MSG_COUNT}"
)
assert len(server_received) == MSG_COUNT, (
f"Server received {len(server_received)} messages, expected {MSG_COUNT}"
)
assert client_received == server_msgs, (
"Client did not receive server messages in order or intact!"
)
assert server_received == client_msgs, (
"Server did not receive client messages in order or intact!"
)
for i, msg in enumerate(client_received):
assert len(msg) == MSG_SIZE, (
f"Client message {i} has wrong size: {len(msg)} != {MSG_SIZE}"
)
for i, msg in enumerate(server_received):
assert len(msg) == MSG_SIZE, (
f"Server message {i} has wrong size: {len(msg)} != {MSG_SIZE}"
)
await client_stream.close()
await server_stream.close()

View File

@ -1,195 +0,0 @@
import logging
import pytest
import trio
from trio.testing import (
memory_stream_pair,
)
from libp2p.abc import IRawConnection
from libp2p.crypto.ed25519 import (
create_new_key_pair,
)
from libp2p.peer.id import (
ID,
)
from libp2p.security.insecure.transport import (
InsecureTransport,
)
from libp2p.stream_muxer.exceptions import MuxedStreamEOF
from libp2p.stream_muxer.yamux.yamux import (
Yamux,
YamuxStream,
)
class TrioStreamAdapter(IRawConnection):
"""Adapter to make trio memory streams work with libp2p."""
def __init__(self, send_stream, receive_stream, is_initiator=False):
self.send_stream = send_stream
self.receive_stream = receive_stream
self.is_initiator = is_initiator
async def write(self, data: bytes) -> None:
logging.debug(f"Attempting to write {len(data)} bytes")
with trio.move_on_after(2):
await self.send_stream.send_all(data)
async def read(self, n: int | None = None) -> bytes:
if n is None or n <= 0:
raise ValueError("Reading unbounded or zero bytes not supported")
logging.debug(f"Attempting to read {n} bytes")
with trio.move_on_after(2):
data = await self.receive_stream.receive_some(n)
logging.debug(f"Read {len(data)} bytes")
return data
async def close(self) -> None:
logging.debug("Closing stream")
await self.send_stream.aclose()
await self.receive_stream.aclose()
def get_remote_address(self) -> tuple[str, int] | None:
"""Return None since this is a test adapter without real network info."""
return None
@pytest.fixture
def key_pair():
return create_new_key_pair()
@pytest.fixture
def peer_id(key_pair):
return ID.from_pubkey(key_pair.public_key)
@pytest.fixture
async def secure_conn_pair(key_pair, peer_id):
"""Create a pair of secure connections for testing."""
logging.debug("Setting up secure_conn_pair")
client_send, server_receive = memory_stream_pair()
server_send, client_receive = memory_stream_pair()
client_rw = TrioStreamAdapter(client_send, client_receive)
server_rw = TrioStreamAdapter(server_send, server_receive)
insecure_transport = InsecureTransport(key_pair)
async def run_outbound(nursery_results):
with trio.move_on_after(5):
client_conn = await insecure_transport.secure_outbound(client_rw, peer_id)
logging.debug("Outbound handshake complete")
nursery_results["client"] = client_conn
async def run_inbound(nursery_results):
with trio.move_on_after(5):
server_conn = await insecure_transport.secure_inbound(server_rw)
logging.debug("Inbound handshake complete")
nursery_results["server"] = server_conn
nursery_results = {}
async with trio.open_nursery() as nursery:
nursery.start_soon(run_outbound, nursery_results)
nursery.start_soon(run_inbound, nursery_results)
await trio.sleep(0.1) # Give tasks a chance to finish
client_conn = nursery_results.get("client")
server_conn = nursery_results.get("server")
if client_conn is None or server_conn is None:
raise RuntimeError("Handshake failed: client_conn or server_conn is None")
logging.debug("secure_conn_pair setup complete")
return client_conn, server_conn
@pytest.fixture
async def yamux_pair(secure_conn_pair, peer_id):
"""Create a pair of Yamux multiplexers for testing."""
logging.debug("Setting up yamux_pair")
client_conn, server_conn = secure_conn_pair
client_yamux = Yamux(client_conn, peer_id, is_initiator=True)
server_yamux = Yamux(server_conn, peer_id, is_initiator=False)
async with trio.open_nursery() as nursery:
with trio.move_on_after(5):
nursery.start_soon(client_yamux.start)
nursery.start_soon(server_yamux.start)
await trio.sleep(0.1)
logging.debug("yamux_pair started")
yield client_yamux, server_yamux
logging.debug("yamux_pair cleanup")
@pytest.mark.trio
async def test_yamux_race_condition_without_locks(yamux_pair):
"""
Test for race-around/interleaving in Yamux streams,when reading till
EOF is being used.
This launches concurrent writers/readers on both sides of a stream.
If there is no proper locking, the received data may be interleaved
or corrupted.
The test creates structured messages and verifies they are received
intact and in order.
Without proper locking, concurrent read/write operations could cause
data corruption
or message interleaving, which this test will catch.
"""
client_yamux, server_yamux = yamux_pair
client_stream: YamuxStream = await client_yamux.open_stream()
server_stream: YamuxStream = await server_yamux.accept_stream()
MSG_COUNT = 1
MSG_SIZE = 512 * 1024
client_msgs = [
f"CLIENT-MSG-{i:03d}-".encode().ljust(MSG_SIZE, b"C") for i in range(MSG_COUNT)
]
server_msgs = [
f"SERVER-MSG-{i:03d}-".encode().ljust(MSG_SIZE, b"S") for i in range(MSG_COUNT)
]
client_received = []
server_received = []
async def writer(stream, msgs, name):
"""Write messages with minimal delays to encourage race conditions."""
for i, msg in enumerate(msgs):
await stream.write(msg)
# Yield control frequently to encourage interleaving
if i % 5 == 0:
await trio.sleep(0.005)
async def reader(stream, received, name):
"""Read messages and store them for verification."""
try:
data = await stream.read()
if data:
received.append(data)
except MuxedStreamEOF:
pass
# Running all operations concurrently
async with trio.open_nursery() as nursery:
nursery.start_soon(writer, client_stream, client_msgs, "client")
nursery.start_soon(writer, server_stream, server_msgs, "server")
nursery.start_soon(reader, client_stream, client_received, "client")
nursery.start_soon(reader, server_stream, server_received, "server")
assert client_received == server_msgs, (
"Client did not receive server messages in order or intact!"
)
assert server_received == client_msgs, (
"Server did not receive client messages in order or intact!"
)
for i, msg in enumerate(client_received):
assert len(msg) == MSG_SIZE, (
f"Client message {i} has wrong size: {len(msg)} != {MSG_SIZE}"
)
for i, msg in enumerate(server_received):
assert len(msg) == MSG_SIZE, (
f"Server message {i} has wrong size: {len(msg)} != {MSG_SIZE}"
)
await client_stream.close()
await server_stream.close()

View File

@ -1,215 +0,0 @@
import pytest
from libp2p.exceptions import ParseError
from libp2p.io.abc import Reader
from libp2p.utils.varint import (
decode_varint_from_bytes,
encode_uvarint,
encode_varint_prefixed,
read_varint_prefixed_bytes,
)
class MockReader(Reader):
"""Mock reader for testing varint functions."""
def __init__(self, data: bytes):
self.data = data
self.position = 0
async def read(self, n: int | None = None) -> bytes:
if self.position >= len(self.data):
return b""
if n is None:
n = len(self.data) - self.position
result = self.data[self.position : self.position + n]
self.position += len(result)
return result
def test_encode_uvarint():
"""Test varint encoding with various values."""
test_cases = [
(0, b"\x00"),
(1, b"\x01"),
(127, b"\x7f"),
(128, b"\x80\x01"),
(255, b"\xff\x01"),
(256, b"\x80\x02"),
(65535, b"\xff\xff\x03"),
(65536, b"\x80\x80\x04"),
(16777215, b"\xff\xff\xff\x07"),
(16777216, b"\x80\x80\x80\x08"),
]
for value, expected in test_cases:
result = encode_uvarint(value)
assert result == expected, (
f"Failed for value {value}: expected {expected.hex()}, got {result.hex()}"
)
def test_decode_varint_from_bytes():
"""Test varint decoding with various values."""
test_cases = [
(b"\x00", 0),
(b"\x01", 1),
(b"\x7f", 127),
(b"\x80\x01", 128),
(b"\xff\x01", 255),
(b"\x80\x02", 256),
(b"\xff\xff\x03", 65535),
(b"\x80\x80\x04", 65536),
(b"\xff\xff\xff\x07", 16777215),
(b"\x80\x80\x80\x08", 16777216),
]
for data, expected in test_cases:
result = decode_varint_from_bytes(data)
assert result == expected, (
f"Failed for data {data.hex()}: expected {expected}, got {result}"
)
def test_decode_varint_from_bytes_invalid():
"""Test varint decoding with invalid data."""
# Empty data
with pytest.raises(ParseError, match="Unexpected end of data"):
decode_varint_from_bytes(b"")
# Incomplete varint (should not raise, but should handle gracefully)
# This depends on the implementation - some might raise, others might return partial
def test_encode_varint_prefixed():
"""Test encoding messages with varint length prefix."""
test_cases = [
(b"", b"\x00"),
(b"hello", b"\x05hello"),
(b"x" * 127, b"\x7f" + b"x" * 127),
(b"x" * 128, b"\x80\x01" + b"x" * 128),
]
for message, expected in test_cases:
result = encode_varint_prefixed(message)
assert result == expected, (
f"Failed for message {message}: expected {expected.hex()}, "
f"got {result.hex()}"
)
@pytest.mark.trio
async def test_read_varint_prefixed_bytes():
"""Test reading length-prefixed bytes from a stream."""
test_cases = [
(b"", b""),
(b"hello", b"hello"),
(b"x" * 127, b"x" * 127),
(b"x" * 128, b"x" * 128),
]
for message, expected in test_cases:
prefixed_data = encode_varint_prefixed(message)
reader = MockReader(prefixed_data)
result = await read_varint_prefixed_bytes(reader)
assert result == expected, (
f"Failed for message {message}: expected {expected}, got {result}"
)
@pytest.mark.trio
async def test_read_varint_prefixed_bytes_incomplete():
"""Test reading length-prefixed bytes with incomplete data."""
from libp2p.io.exceptions import IncompleteReadError
# Test with incomplete varint
reader = MockReader(b"\x80") # Incomplete varint
with pytest.raises(IncompleteReadError):
await read_varint_prefixed_bytes(reader)
# Test with incomplete message
prefixed_data = encode_varint_prefixed(b"hello world")
reader = MockReader(prefixed_data[:-3]) # Missing last 3 bytes
with pytest.raises(IncompleteReadError):
await read_varint_prefixed_bytes(reader)
def test_varint_roundtrip():
"""Test roundtrip encoding and decoding."""
test_values = [0, 1, 127, 128, 255, 256, 65535, 65536, 16777215, 16777216]
for value in test_values:
encoded = encode_uvarint(value)
decoded = decode_varint_from_bytes(encoded)
assert decoded == value, (
f"Roundtrip failed for {value}: encoded={encoded.hex()}, decoded={decoded}"
)
def test_varint_prefixed_roundtrip():
"""Test roundtrip encoding and decoding of length-prefixed messages."""
test_messages = [
b"",
b"hello",
b"x" * 127,
b"x" * 128,
b"x" * 1000,
]
for message in test_messages:
prefixed = encode_varint_prefixed(message)
# Decode the length
length = decode_varint_from_bytes(prefixed)
assert length == len(message), (
f"Length mismatch for {message}: expected {len(message)}, got {length}"
)
# Extract the message
varint_len = 0
for i, byte in enumerate(prefixed):
varint_len += 1
if (byte & 0x80) == 0:
break
extracted_message = prefixed[varint_len:]
assert extracted_message == message, (
f"Message mismatch: expected {message}, got {extracted_message}"
)
def test_large_varint_values():
"""Test varint encoding/decoding with large values."""
large_values = [
2**32 - 1, # 32-bit max
2**64 - 1, # 64-bit max (if supported)
]
for value in large_values:
try:
encoded = encode_uvarint(value)
decoded = decode_varint_from_bytes(encoded)
assert decoded == value, f"Large value roundtrip failed for {value}"
except Exception as e:
# Some implementations might not support very large values
pytest.skip(f"Large value {value} not supported: {e}")
def test_varint_edge_cases():
"""Test varint encoding/decoding with edge cases."""
# Test with maximum 7-bit value
assert encode_uvarint(127) == b"\x7f"
assert decode_varint_from_bytes(b"\x7f") == 127
# Test with minimum 8-bit value
assert encode_uvarint(128) == b"\x80\x01"
assert decode_varint_from_bytes(b"\x80\x01") == 128
# Test with maximum 14-bit value
assert encode_uvarint(16383) == b"\xff\x7f"
assert decode_varint_from_bytes(b"\xff\x7f") == 16383
# Test with minimum 15-bit value
assert encode_uvarint(16384) == b"\x80\x80\x01"
assert decode_varint_from_bytes(b"\x80\x80\x01") == 16384

View File

@ -1,91 +0,0 @@
"""
Unit tests for mDNS broadcaster component.
"""
from zeroconf import Zeroconf
from libp2p.discovery.mdns.broadcaster import PeerBroadcaster
from libp2p.peer.id import ID
class TestPeerBroadcaster:
"""Unit tests for PeerBroadcaster."""
def test_broadcaster_initialization(self):
"""Test that broadcaster initializes correctly."""
zeroconf = Zeroconf()
service_type = "_p2p._udp.local."
service_name = "test-peer._p2p._udp.local."
peer_id = (
"QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN" # String, not ID object
)
port = 8000
broadcaster = PeerBroadcaster(
zeroconf=zeroconf,
service_type=service_type,
service_name=service_name,
peer_id=peer_id,
port=port,
)
assert broadcaster.zeroconf == zeroconf
assert broadcaster.service_type == service_type
assert broadcaster.service_name == service_name
assert broadcaster.peer_id == peer_id
assert broadcaster.port == port
# Clean up
zeroconf.close()
def test_broadcaster_service_creation(self):
"""Test that broadcaster creates valid service info."""
zeroconf = Zeroconf()
service_type = "_p2p._udp.local."
service_name = "test-peer2._p2p._udp.local."
peer_id_obj = ID.from_base58("QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN")
peer_id = str(peer_id_obj) # Convert to string
port = 8000
broadcaster = PeerBroadcaster(
zeroconf=zeroconf,
service_type=service_type,
service_name=service_name,
peer_id=peer_id,
port=port,
)
# Verify service was created and registered
service_info = broadcaster.service_info
assert service_info is not None
assert service_info.type == service_type
assert service_info.name == service_name
assert service_info.port == port
assert b"id" in service_info.properties
assert service_info.properties[b"id"] == peer_id.encode()
# Clean up
zeroconf.close()
def test_broadcaster_start_stop(self):
"""Test that broadcaster can start and stop correctly."""
zeroconf = Zeroconf()
service_type = "_p2p._udp.local."
service_name = "test-start-stop._p2p._udp.local."
peer_id_obj = ID.from_base58("QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N")
peer_id = str(peer_id_obj) # Convert to string
port = 8001
broadcaster = PeerBroadcaster(
zeroconf=zeroconf,
service_type=service_type,
service_name=service_name,
peer_id=peer_id,
port=port,
)
# Service should be registered
assert broadcaster.service_info is not None
# Clean up
zeroconf.close()

View File

@ -1,114 +0,0 @@
"""
Unit tests for mDNS listener component.
"""
import socket
from zeroconf import ServiceInfo, Zeroconf
from libp2p.abc import Multiaddr
from libp2p.discovery.mdns.listener import PeerListener
from libp2p.peer.id import ID
from libp2p.peer.peerstore import PeerStore
class TestPeerListener:
"""Unit tests for PeerListener."""
def test_listener_initialization(self):
"""Test that listener initializes correctly."""
peerstore = PeerStore()
zeroconf = Zeroconf()
service_type = "_p2p._udp.local."
service_name = "local-peer._p2p._udp.local."
listener = PeerListener(
peerstore=peerstore,
zeroconf=zeroconf,
service_type=service_type,
service_name=service_name,
)
assert listener.peerstore == peerstore
assert listener.zeroconf == zeroconf
assert listener.service_type == service_type
assert listener.service_name == service_name
assert listener.discovered_services == {}
# Clean up
listener.stop()
zeroconf.close()
def test_listener_extract_peer_info_success(self):
"""Test successful PeerInfo extraction from ServiceInfo."""
peerstore = PeerStore()
zeroconf = Zeroconf()
listener = PeerListener(
peerstore=peerstore,
zeroconf=zeroconf,
service_type="_p2p._udp.local.",
service_name="local._p2p._udp.local.",
)
# Create sample service info
sample_peer_id = ID.from_base58(
"QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN"
)
hostname = socket.gethostname()
local_ip = "192.168.1.100"
sample_service_info = ServiceInfo(
type_="_p2p._udp.local.",
name="test-peer._p2p._udp.local.",
port=8000,
properties={b"id": str(sample_peer_id).encode()},
server=f"{hostname}.local.",
addresses=[socket.inet_aton(local_ip)],
)
peer_info = listener._extract_peer_info(sample_service_info)
assert peer_info is not None
assert isinstance(peer_info.peer_id, ID)
assert len(peer_info.addrs) > 0
assert all(isinstance(addr, Multiaddr) for addr in peer_info.addrs)
# Check that protocol is TCP since we always use TCP
assert "/tcp/" in str(peer_info.addrs[0])
# Clean up
listener.stop()
zeroconf.close()
def test_listener_extract_peer_info_invalid_id(self):
"""Test PeerInfo extraction fails with invalid peer ID."""
peerstore = PeerStore()
zeroconf = Zeroconf()
listener = PeerListener(
peerstore=peerstore,
zeroconf=zeroconf,
service_type="_p2p._udp.local.",
service_name="local._p2p._udp.local.",
)
# Create service info with invalid peer ID
hostname = socket.gethostname()
local_ip = "192.168.1.100"
service_info = ServiceInfo(
type_="_p2p._udp.local.",
name="invalid-peer._p2p._udp.local.",
port=8000,
properties={b"id": b"invalid_peer_id_format"},
server=f"{hostname}.local.",
addresses=[socket.inet_aton(local_ip)],
)
peer_info = listener._extract_peer_info(service_info)
assert peer_info is None
# Clean up
listener.stop()
zeroconf.close()

View File

@ -1,121 +0,0 @@
"""
Comprehensive integration tests for mDNS discovery functionality.
"""
import socket
from zeroconf import Zeroconf
from libp2p.discovery.mdns.broadcaster import PeerBroadcaster
from libp2p.discovery.mdns.listener import PeerListener
from libp2p.peer.id import ID
from libp2p.peer.peerstore import PeerStore
class TestMDNSDiscovery:
"""Comprehensive integration tests for mDNS peer discovery."""
def test_one_host_finds_another(self):
"""Test that one host can find another host using mDNS."""
# Create two separate Zeroconf instances to simulate different hosts
host1_zeroconf = Zeroconf()
host2_zeroconf = Zeroconf()
try:
# Host 1: Set up as broadcaster (the host to be discovered)
host1_peer_id_obj = ID.from_base58(
"QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN"
)
host1_peer_id = str(host1_peer_id_obj) # Convert to string
host1_broadcaster = PeerBroadcaster(
zeroconf=host1_zeroconf,
service_type="_p2p._udp.local.",
service_name="host1._p2p._udp.local.",
peer_id=host1_peer_id,
port=8000,
)
# Host 2: Set up as listener (the host that discovers others)
host2_peerstore = PeerStore()
host2_listener = PeerListener(
peerstore=host2_peerstore,
zeroconf=host2_zeroconf,
service_type="_p2p._udp.local.",
service_name="host2._p2p._udp.local.",
)
# Host 1 registers its service for discovery
host1_broadcaster.register()
# Verify that host2 discovered host1
assert len(host2_listener.discovered_services) > 0
assert "host1._p2p._udp.local." in host2_listener.discovered_services
# Verify that host1's peer info was added to host2's peerstore
discovered_peer_id = host2_listener.discovered_services[
"host1._p2p._udp.local."
]
assert str(discovered_peer_id) == host1_peer_id
# Verify addresses were added to peerstore
try:
addrs = host2_peerstore.addrs(discovered_peer_id)
assert len(addrs) > 0
# Should be TCP since we always use TCP protocol
assert "/tcp/8000" in str(addrs[0])
except Exception:
# If no addresses found, the discovery didn't work properly
assert False, "Host1 addresses should be in Host2's peerstore"
# Clean up
host1_broadcaster.unregister()
host2_listener.stop()
finally:
host1_zeroconf.close()
host2_zeroconf.close()
def test_service_info_extraction(self):
"""Test service info extraction functionality."""
peerstore = PeerStore()
zeroconf = Zeroconf()
try:
listener = PeerListener(
peerstore=peerstore,
zeroconf=zeroconf,
service_type="_p2p._udp.local.",
service_name="test-listener._p2p._udp.local.",
)
# Create a test service info
test_peer_id = ID.from_base58(
"QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"
)
hostname = socket.gethostname()
from zeroconf import ServiceInfo
service_info = ServiceInfo(
type_="_p2p._udp.local.",
name="test-service._p2p._udp.local.",
port=8001,
properties={b"id": str(test_peer_id).encode()},
server=f"{hostname}.local.",
addresses=[socket.inet_aton("192.168.1.100")],
)
# Test extraction
peer_info = listener._extract_peer_info(service_info)
assert peer_info is not None
assert peer_info.peer_id == test_peer_id
assert len(peer_info.addrs) == 1
assert "/tcp/8001" in str(peer_info.addrs[0])
print("✅ Service info extraction test successful!")
print(f" Extracted peer ID: {peer_info.peer_id}")
print(f" Extracted addresses: {[str(addr) for addr in peer_info.addrs]}")
finally:
zeroconf.close()

View File

@ -1,39 +0,0 @@
"""
Basic unit tests for mDNS utils module.
"""
import string
from libp2p.discovery.mdns.utils import stringGen
class TestStringGen:
"""Unit tests for stringGen function."""
def test_stringgen_default_length(self):
"""Test stringGen with default length (63)."""
result = stringGen()
assert isinstance(result, str)
assert len(result) == 63
# Check that all characters are from the expected charset
charset = string.ascii_lowercase + string.digits
for char in result:
assert char in charset
def test_stringgen_custom_length(self):
"""Test stringGen with custom lengths."""
# Test various lengths
test_lengths = [1, 5, 10, 20, 50, 100]
for length in test_lengths:
result = stringGen(length)
assert isinstance(result, str)
assert len(result) == length
# Check that all characters are from the expected charset
charset = string.ascii_lowercase + string.digits
for char in result:
assert char in charset

View File

@ -1,81 +0,0 @@
# py-libp2p and js-libp2p Interoperability Tests
This repository contains interoperability tests for py-libp2p and js-libp2p using the /ipfs/ping/1.0.0 protocol. The goal is to verify compatibility in stream multiplexing, protocol negotiation, ping handling, transport layer, and multiaddr parsing.
## Directory Structure
- js_node/ping.js: JavaScript implementation of a ping server and client using libp2p.
- py_node/ping.py: Python implementation of a ping server and client using py-libp2p.
- scripts/run_test.sh: Shell script to automate running the server and client for testing.
- README.md: This file.
## Prerequisites
- Python 3.8+ with `py-libp2p` and dependencies (`pip install libp2p trio cryptography multiaddr`).
- Node.js 16+ with `libp2p` dependencies (`npm install @libp2p/core @libp2p/tcp @chainsafe/libp2p-noise @chainsafe/libp2p-yamux @libp2p/ping @libp2p/identify @multiformats/multiaddr`).
- Bash shell for running `run_test.sh`.
## Running Tests
1. Change directory:
```
cd tests/interop/js_libp2p
```
2. Install dependencies:
```
For JavaScript: cd js_node && npm install && cd ...
```
3. Run the automated test:
For Linux and Mac users:
```
chmod +x scripts/run_test.sh
./scripts/run_test.sh
```
For Windows users:
```
.\scripts\run_test.ps1
```
This starts the Python server on port 8000 and runs the JavaScript client to send 5 pings.
## Debugging
- Logs are saved in py_node/py_server.log and js_node/js_client.log.
- Check for:
- Successful connection establishment.
- Protocol negotiation (/ipfs/ping/1.0.0).
- 32-byte payload echo in server logs.
- RTT and payload hex in client logs.
## Test Plan
### The test verifies:
- Stream Multiplexer Compatibility: Yamux is used and negotiates correctly.
- Multistream Protocol Negotiation: /ipfs/ping/1.0.0 is selected via multistream-select.
- Ping Protocol Handler: Handles 32-byte payloads per the libp2p ping spec.
- Transport Layer Support: TCP is used; WebSocket support is optional.
- Multiaddr Parsing: Correctly resolves multiaddr strings.
- Logging: Includes peer ID, RTT, and payload hex for debugging.
## Current Status
### Working:
- TCP transport and Noise encryption are functional.
- Yamux multiplexing is implemented in both nodes.
- Multiaddr parsing works correctly.
- Logging provides detailed debug information.
## Not Working:
- Ping protocol handler fails to complete pings (JS client reports "operation aborted").
- Potential issues with stream handling or protocol negotiation.

View File

@ -1,53 +0,0 @@
# @libp2p/example-chat <!-- omit in toc -->
[![libp2p.io](https://img.shields.io/badge/project-libp2p-yellow.svg?style=flat-square)](http://libp2p.io/)
[![Discuss](https://img.shields.io/discourse/https/discuss.libp2p.io/posts.svg?style=flat-square)](https://discuss.libp2p.io)
[![codecov](https://img.shields.io/codecov/c/github/libp2p/js-libp2p-examples.svg?style=flat-square)](https://codecov.io/gh/libp2p/js-libp2p-examples)
[![CI](https://img.shields.io/github/actions/workflow/status/libp2p/js-libp2p-examples/ci.yml?branch=main&style=flat-square)](https://github.com/libp2p/js-libp2p-examples/actions/workflows/ci.yml?query=branch%3Amain)
> An example chat app using libp2p
## Table of contents <!-- omit in toc -->
- [Setup](#setup)
- [Running](#running)
- [Need help?](#need-help)
- [License](#license)
- [Contribution](#contribution)
## Setup
1. Install example dependencies
```console
$ npm install
```
1. Open 2 terminal windows in the `./src` directory.
## Running
1. Run the listener in window 1, `node listener.js`
1. Run the dialer in window 2, `node dialer.js`
1. Wait until the two peers discover each other
1. Type a message in either window and hit *enter*
1. Tell yourself secrets to your hearts content!
## Need help?
- Read the [js-libp2p documentation](https://github.com/libp2p/js-libp2p/tree/main/doc)
- Check out the [js-libp2p API docs](https://libp2p.github.io/js-libp2p/)
- Check out the [general libp2p documentation](https://docs.libp2p.io) for tips, how-tos and more
- Read the [libp2p specs](https://github.com/libp2p/specs)
- Ask a question on the [js-libp2p discussion board](https://github.com/libp2p/js-libp2p/discussions)
## License
Licensed under either of
- Apache 2.0, ([LICENSE-APACHE](LICENSE-APACHE) / <http://www.apache.org/licenses/LICENSE-2.0>)
- MIT ([LICENSE-MIT](LICENSE-MIT) / <http://opensource.org/licenses/MIT>)
## Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
dual licensed as above, without any additional terms or conditions.

View File

@ -1,39 +0,0 @@
{
"name": "@libp2p/example-chat",
"version": "0.0.0",
"description": "An example chat app using libp2p",
"license": "Apache-2.0 OR MIT",
"homepage": "https://github.com/libp2p/js-libp2p-example-chat#readme",
"repository": {
"type": "git",
"url": "git+https://github.com/libp2p/js-libp2p-examples.git"
},
"bugs": {
"url": "https://github.com/libp2p/js-libp2p-examples/issues"
},
"type": "module",
"scripts": {
"test": "test-node-example test/*"
},
"dependencies": {
"@chainsafe/libp2p-noise": "^16.0.0",
"@chainsafe/libp2p-yamux": "^7.0.0",
"@libp2p/identify": "^3.0.33",
"@libp2p/mdns": "^11.0.1",
"@libp2p/ping": "^2.0.33",
"@libp2p/tcp": "^10.0.0",
"@libp2p/websockets": "^9.0.0",
"@multiformats/multiaddr": "^12.3.1",
"@nodeutils/defaults-deep": "^1.1.0",
"it-length-prefixed": "^10.0.1",
"it-map": "^3.0.3",
"it-pipe": "^3.0.1",
"libp2p": "^2.0.0",
"p-defer": "^4.0.0",
"uint8arrays": "^5.1.0"
},
"devDependencies": {
"test-ipfs-example": "^1.1.0"
},
"private": true
}

View File

@ -1,204 +0,0 @@
#!/usr/bin/env node
import { createLibp2p } from 'libp2p'
import { tcp } from '@libp2p/tcp'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { ping } from '@libp2p/ping'
import { identify } from '@libp2p/identify'
import { multiaddr } from '@multiformats/multiaddr'
async function createNode() {
return await createLibp2p({
addresses: {
listen: ['/ip4/0.0.0.0/tcp/0']
},
transports: [
tcp()
],
connectionEncrypters: [
noise()
],
streamMuxers: [
yamux()
],
services: {
// Use ipfs prefix to match py-libp2p example
ping: ping({
protocolPrefix: 'ipfs',
maxInboundStreams: 32,
maxOutboundStreams: 64,
timeout: 30000
}),
identify: identify()
},
connectionManager: {
minConnections: 0,
maxConnections: 100,
dialTimeout: 30000
}
})
}
async function runServer() {
console.log('🚀 Starting js-libp2p ping server...')
const node = await createNode()
await node.start()
console.log('✅ Server started!')
console.log(`📋 Peer ID: ${node.peerId.toString()}`)
console.log('📍 Listening addresses:')
node.getMultiaddrs().forEach(addr => {
console.log(` ${addr.toString()}`)
})
// Listen for connections
node.addEventListener('peer:connect', (evt) => {
console.log(`🔗 Peer connected: ${evt.detail.toString()}`)
})
node.addEventListener('peer:disconnect', (evt) => {
console.log(`❌ Peer disconnected: ${evt.detail.toString()}`)
})
console.log('\n🎧 Server ready for ping requests...')
console.log('Press Ctrl+C to exit')
// Graceful shutdown
process.on('SIGINT', async () => {
console.log('\n🛑 Shutting down...')
await node.stop()
process.exit(0)
})
// Keep alive
while (true) {
await new Promise(resolve => setTimeout(resolve, 1000))
}
}
async function runClient(targetAddr, count = 5) {
console.log('🚀 Starting js-libp2p ping client...')
const node = await createNode()
await node.start()
console.log(`📋 Our Peer ID: ${node.peerId.toString()}`)
console.log(`🎯 Target: ${targetAddr}`)
try {
const ma = multiaddr(targetAddr)
const targetPeerId = ma.getPeerId()
if (!targetPeerId) {
throw new Error('Could not extract peer ID from multiaddr')
}
console.log(`🎯 Target Peer ID: ${targetPeerId}`)
console.log('🔗 Connecting to peer...')
const connection = await node.dial(ma)
console.log('✅ Connection established!')
console.log(`🔗 Connected to: ${connection.remotePeer.toString()}`)
// Add a small delay to let the connection fully establish
await new Promise(resolve => setTimeout(resolve, 1000))
const rtts = []
for (let i = 1; i <= count; i++) {
try {
console.log(`\n🏓 Sending ping ${i}/${count}...`);
console.log('[DEBUG] Attempting to open ping stream with protocol: /ipfs/ping/1.0.0');
const start = Date.now()
const stream = await connection.newStream(['/ipfs/ping/1.0.0']).catch(err => {
console.error(`[ERROR] Failed to open ping stream: ${err.message}`);
throw err;
});
console.log('[DEBUG] Ping stream opened successfully');
const latency = await Promise.race([
node.services.ping.ping(connection.remotePeer),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('Ping timeout')), 30000) // Increased timeout
)
]).catch(err => {
console.error(`[ERROR] Ping ${i} error: ${err.message}`);
throw err;
});
const rtt = Date.now() - start;
rtts.push(latency)
console.log(`✅ Ping ${i} successful!`)
console.log(` Reported latency: ${latency}ms`)
console.log(` Measured RTT: ${rtt}ms`)
if (i < count) {
await new Promise(resolve => setTimeout(resolve, 1000))
}
} catch (error) {
console.error(`❌ Ping ${i} failed:`, error.message)
// Try to continue with other pings
}
}
// Stats
if (rtts.length > 0) {
const avg = rtts.reduce((a, b) => a + b, 0) / rtts.length
const min = Math.min(...rtts)
const max = Math.max(...rtts)
console.log(`\n📊 Ping Statistics:`)
console.log(` Packets: Sent=${count}, Received=${rtts.length}, Lost=${count - rtts.length}`)
console.log(` Latency: min=${min}ms, avg=${avg.toFixed(2)}ms, max=${max}ms`)
} else {
console.log(`\n📊 All pings failed (${count} attempts)`)
}
} catch (error) {
console.error('❌ Client error:', error.message)
console.error('Stack:', error.stack)
process.exit(1)
} finally {
await node.stop()
console.log('\n⏹ Client stopped')
}
}
async function main() {
const args = process.argv.slice(2)
if (args.length === 0) {
console.log('Usage:')
console.log(' node ping.js server # Start ping server')
console.log(' node ping.js client <multiaddr> [count] # Ping a peer')
console.log('')
console.log('Examples:')
console.log(' node ping.js server')
console.log(' node ping.js client /ip4/127.0.0.1/tcp/12345/p2p/12D3Ko... 5')
process.exit(1)
}
const mode = args[0]
if (mode === 'server') {
await runServer()
} else if (mode === 'client') {
if (args.length < 2) {
console.error('❌ Client mode requires target multiaddr')
process.exit(1)
}
const targetAddr = args[1]
const count = parseInt(args[2]) || 5
await runClient(targetAddr, count)
} else {
console.error('❌ Invalid mode. Use "server" or "client"')
process.exit(1)
}
}
main().catch(console.error)

View File

@ -1,241 +0,0 @@
#!/usr/bin/env node
import { createLibp2p } from 'libp2p'
import { tcp } from '@libp2p/tcp'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { ping } from '@libp2p/ping'
import { identify } from '@libp2p/identify'
import { multiaddr } from '@multiformats/multiaddr'
import fs from 'fs'
import path from 'path'
// Create logs directory if it doesn't exist
const logsDir = path.join(process.cwd(), '../logs')
if (!fs.existsSync(logsDir)) {
fs.mkdirSync(logsDir, { recursive: true })
}
// Setup logging
const logFile = path.join(logsDir, 'js_ping_client.log')
const logStream = fs.createWriteStream(logFile, { flags: 'w' })
function log(message) {
const timestamp = new Date().toISOString()
const logLine = `${timestamp} - ${message}\n`
logStream.write(logLine)
console.log(message)
}
async function createNode() {
log('🔧 Creating libp2p node...')
const node = await createLibp2p({
addresses: {
listen: ['/ip4/0.0.0.0/tcp/0'] // Random port
},
transports: [
tcp()
],
connectionEncrypters: [
noise()
],
streamMuxers: [
yamux()
],
services: {
ping: ping({
protocolPrefix: 'ipfs', // Use ipfs prefix to match py-libp2p
maxInboundStreams: 32,
maxOutboundStreams: 64,
timeout: 30000,
runOnTransientConnection: true
}),
identify: identify()
},
connectionManager: {
minConnections: 0,
maxConnections: 100,
dialTimeout: 30000,
maxParallelDials: 10
}
})
log('✅ Node created successfully')
return node
}
async function runClient(targetAddr, count = 5) {
log('🚀 Starting js-libp2p ping client...')
const node = await createNode()
// Add connection event listeners
node.addEventListener('peer:connect', (evt) => {
log(`🔗 Connected to peer: ${evt.detail.toString()}`)
})
node.addEventListener('peer:disconnect', (evt) => {
log(`❌ Disconnected from peer: ${evt.detail.toString()}`)
})
await node.start()
log('✅ Node started')
log(`📋 Our Peer ID: ${node.peerId.toString()}`)
log(`🎯 Target: ${targetAddr}`)
try {
const ma = multiaddr(targetAddr)
const targetPeerId = ma.getPeerId()
if (!targetPeerId) {
throw new Error('Could not extract peer ID from multiaddr')
}
log(`🎯 Target Peer ID: ${targetPeerId}`)
// Parse multiaddr components for debugging
const components = ma.toString().split('/')
log(`📍 Target components: ${components.join(' → ')}`)
log('🔗 Attempting to dial peer...')
const connection = await node.dial(ma)
log('✅ Connection established!')
log(`🔗 Connected to: ${connection.remotePeer.toString()}`)
log(`🔗 Connection status: ${connection.status}`)
log(`🔗 Connection direction: ${connection.direction}`)
// List available protocols
if (connection.remoteAddr) {
log(`🌐 Remote address: ${connection.remoteAddr.toString()}`)
}
// Wait for connection to stabilize
log('⏳ Waiting for connection to stabilize...')
await new Promise(resolve => setTimeout(resolve, 2000))
// Attempt ping sequence
log(`\n🏓 Starting ping sequence (${count} pings)...`)
const rtts = []
for (let i = 1; i <= count; i++) {
try {
log(`\n🏓 Sending ping ${i}/${count}...`)
const start = Date.now()
// Create a more robust ping with better error handling
const pingPromise = node.services.ping.ping(connection.remotePeer)
const timeoutPromise = new Promise((_, reject) =>
setTimeout(() => reject(new Error('Ping timeout (15s)')), 15000)
)
const latency = await Promise.race([pingPromise, timeoutPromise])
const totalRtt = Date.now() - start
rtts.push(latency)
log(`✅ Ping ${i} successful!`)
log(` Reported latency: ${latency}ms`)
log(` Total RTT: ${totalRtt}ms`)
// Wait between pings
if (i < count) {
await new Promise(resolve => setTimeout(resolve, 1000))
}
} catch (error) {
log(`❌ Ping ${i} failed: ${error.message}`)
log(` Error type: ${error.constructor.name}`)
if (error.code) {
log(` Error code: ${error.code}`)
}
// Check if connection is still alive
if (connection.status !== 'open') {
log(`⚠️ Connection status changed to: ${connection.status}`)
break
}
}
}
// Print statistics
if (rtts.length > 0) {
const avg = rtts.reduce((a, b) => a + b, 0) / rtts.length
const min = Math.min(...rtts)
const max = Math.max(...rtts)
const lossRate = ((count - rtts.length) / count * 100).toFixed(1)
log(`\n📊 Ping Statistics:`)
log(` Packets: Sent=${count}, Received=${rtts.length}, Lost=${count - rtts.length}`)
log(` Loss rate: ${lossRate}%`)
log(` Latency: min=${min}ms, avg=${avg.toFixed(2)}ms, max=${max}ms`)
} else {
log(`\n📊 All pings failed (${count} attempts)`)
}
// Close connection gracefully
log('\n🔒 Closing connection...')
await connection.close()
} catch (error) {
log(`❌ Client error: ${error.message}`)
log(` Error type: ${error.constructor.name}`)
if (error.stack) {
log(` Stack trace: ${error.stack}`)
}
process.exit(1)
} finally {
log('🛑 Stopping node...')
await node.stop()
log('⏹️ Client stopped')
logStream.end()
}
}
async function main() {
const args = process.argv.slice(2)
if (args.length === 0) {
console.log('Usage:')
console.log(' node ping-client.js <target-multiaddr> [count]')
console.log('')
console.log('Examples:')
console.log(' node ping-client.js /ip4/127.0.0.1/tcp/8000/p2p/QmExample... 5')
console.log(' node ping-client.js /ip4/127.0.0.1/tcp/8000/p2p/QmExample... 10')
process.exit(1)
}
const targetAddr = args[0]
const count = parseInt(args[1]) || 5
if (count <= 0 || count > 100) {
console.error('❌ Count must be between 1 and 100')
process.exit(1)
}
await runClient(targetAddr, count)
}
// Handle graceful shutdown
process.on('SIGINT', () => {
log('\n👋 Shutting down...')
logStream.end()
process.exit(0)
})
process.on('uncaughtException', (error) => {
log(`💥 Uncaught exception: ${error.message}`)
if (error.stack) {
log(`Stack: ${error.stack}`)
}
logStream.end()
process.exit(1)
})
main().catch((error) => {
log(`💥 Fatal error: ${error.message}`)
if (error.stack) {
log(`Stack: ${error.stack}`)
}
logStream.end()
process.exit(1)
})

View File

@ -1,167 +0,0 @@
#!/usr/bin/env node
import { createLibp2p } from 'libp2p'
import { tcp } from '@libp2p/tcp'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { ping } from '@libp2p/ping'
import { identify } from '@libp2p/identify'
import fs from 'fs'
import path from 'path'
// Create logs directory if it doesn't exist
const logsDir = path.join(process.cwd(), '../logs')
if (!fs.existsSync(logsDir)) {
fs.mkdirSync(logsDir, { recursive: true })
}
// Setup logging
const logFile = path.join(logsDir, 'js_ping_server.log')
const logStream = fs.createWriteStream(logFile, { flags: 'w' })
function log(message) {
const timestamp = new Date().toISOString()
const logLine = `${timestamp} - ${message}\n`
logStream.write(logLine)
console.log(message)
}
async function createNode(port) {
log('🔧 Creating libp2p node...')
const node = await createLibp2p({
addresses: {
listen: [`/ip4/0.0.0.0/tcp/${port}`]
},
transports: [
tcp()
],
connectionEncrypters: [
noise()
],
streamMuxers: [
yamux()
],
services: {
ping: ping({
protocolPrefix: 'ipfs', // Use ipfs prefix to match py-libp2p
maxInboundStreams: 32,
maxOutboundStreams: 64,
timeout: 30000,
runOnTransientConnection: true
}),
identify: identify()
},
connectionManager: {
minConnections: 0,
maxConnections: 100,
dialTimeout: 30000,
maxParallelDials: 10
}
})
log('✅ Node created successfully')
return node
}
async function runServer(port) {
log('🚀 Starting js-libp2p ping server...')
const node = await createNode(port)
// Add connection event listeners
node.addEventListener('peer:connect', (evt) => {
log(`🔗 New peer connected: ${evt.detail.toString()}`)
})
node.addEventListener('peer:disconnect', (evt) => {
log(`❌ Peer disconnected: ${evt.detail.toString()}`)
})
// Add protocol handler for incoming streams
node.addEventListener('peer:identify', (evt) => {
log(`🔍 Peer identified: ${evt.detail.peerId.toString()}`)
log(` Protocols: ${evt.detail.protocols.join(', ')}`)
log(` Listen addresses: ${evt.detail.listenAddrs.map(addr => addr.toString()).join(', ')}`)
})
await node.start()
log('✅ Node started')
const peerId = node.peerId.toString()
const listenAddrs = node.getMultiaddrs()
log(`📋 Peer ID: ${peerId}`)
log(`🌐 Listen addresses:`)
listenAddrs.forEach(addr => {
log(` ${addr.toString()}`)
})
// Find the main TCP address for easy copy-paste
const tcpAddr = listenAddrs.find(addr =>
addr.toString().includes('/tcp/') &&
!addr.toString().includes('/ws')
)
if (tcpAddr) {
log(`\n🧪 Test with py-libp2p:`)
log(` python ping_client.py ${tcpAddr.toString()}`)
log(`\n🧪 Test with js-libp2p:`)
log(` node ping-client.js ${tcpAddr.toString()}`)
}
log(`\n🏓 Ping service is running with protocol: /ipfs/ping/1.0.0`)
log(`🔐 Security: Noise encryption`)
log(`🚇 Muxer: Yamux stream multiplexing`)
log(`\n⏳ Waiting for connections...`)
log('Press Ctrl+C to exit')
// Keep the server running
return new Promise((resolve, reject) => {
process.on('SIGINT', () => {
log('\n🛑 Shutting down server...')
node.stop().then(() => {
log('⏹️ Server stopped')
logStream.end()
resolve()
}).catch(reject)
})
process.on('uncaughtException', (error) => {
log(`💥 Uncaught exception: ${error.message}`)
if (error.stack) {
log(`Stack: ${error.stack}`)
}
logStream.end()
reject(error)
})
})
}
async function main() {
const args = process.argv.slice(2)
const port = parseInt(args[0]) || 9000
if (port <= 0 || port > 65535) {
console.error('❌ Port must be between 1 and 65535')
process.exit(1)
}
try {
await runServer(port)
} catch (error) {
console.error(`💥 Fatal error: ${error.message}`)
if (error.stack) {
console.error(`Stack: ${error.stack}`)
}
process.exit(1)
}
}
main().catch((error) => {
console.error(`💥 Fatal error: ${error.message}`)
if (error.stack) {
console.error(`Stack: ${error.stack}`)
}
process.exit(1)
})

View File

@ -1,194 +0,0 @@
#!/usr/bin/env pwsh
# run_test.ps1 - libp2p Interoperability Test Runner (PowerShell)
# Tests py-libp2p <-> js-libp2p ping communication
$ErrorActionPreference = "Stop"
# Colors for output
$Red = "`e[31m"
$Green = "`e[32m"
$Yellow = "`e[33m"
$Blue = "`e[34m"
$Cyan = "`e[36m"
$Reset = "`e[0m"
function Write-ColorOutput {
param([string]$Message, [string]$Color = $Reset)
Write-Host "${Color}${Message}${Reset}"
}
Write-ColorOutput "[CHECK] Checking prerequisites..." $Cyan
if (-not (Get-Command python -ErrorAction SilentlyContinue)) {
Write-ColorOutput "[ERROR] Python not found. Install Python 3.7+" $Red
exit 1
}
if (-not (Get-Command node -ErrorAction SilentlyContinue)) {
Write-ColorOutput "[ERROR] Node.js not found. Install Node.js 16+" $Red
exit 1
}
Write-ColorOutput "[CHECK] Checking port 8000..." $Blue
$portCheck = netstat -a -n -o | findstr :8000
if ($portCheck) {
Write-ColorOutput "[ERROR] Port 8000 in use. Free the port." $Red
Write-ColorOutput $portCheck $Yellow
exit 1
}
Write-ColorOutput "[DEBUG] Cleaning up Python processes..." $Blue
Get-Process -Name "python" -ErrorAction SilentlyContinue | Where-Object { $_.CommandLine -like "*ping.py*" } | Stop-Process -Force -ErrorAction SilentlyContinue
Write-ColorOutput "[PYTHON] Starting server on port 8000..." $Yellow
Set-Location -Path "py_node"
$pyLogFile = "py_server_8000.log"
$pyErrLogFile = "py_server_8000.log.err"
$pyDebugLogFile = "ping_debug.log"
if (Test-Path $pyLogFile) { Remove-Item $pyLogFile -Force -ErrorAction SilentlyContinue }
if (Test-Path $pyErrLogFile) { Remove-Item $pyErrLogFile -Force -ErrorAction SilentlyContinue }
if (Test-Path $pyDebugLogFile) { Remove-Item $pyDebugLogFile -Force -ErrorAction SilentlyContinue }
$pyProcess = Start-Process -FilePath "python" -ArgumentList "-u", "ping.py", "server", "--port", "8000" -NoNewWindow -PassThru -RedirectStandardOutput $pyLogFile -RedirectStandardError $pyErrLogFile
Write-ColorOutput "[DEBUG] Python server PID: $($pyProcess.Id)" $Blue
Write-ColorOutput "[DEBUG] Python logs: $((Get-Location).Path)\$pyLogFile, $((Get-Location).Path)\$pyErrLogFile, $((Get-Location).Path)\$pyDebugLogFile" $Blue
$timeoutSeconds = 20
$startTime = Get-Date
$serverStarted = $false
while (((Get-Date) - $startTime).TotalSeconds -lt $timeoutSeconds -and -not $serverStarted) {
if (Test-Path $pyLogFile) {
$content = Get-Content $pyLogFile -Raw -ErrorAction SilentlyContinue
if ($content -match "Server started|Listening") {
$serverStarted = $true
Write-ColorOutput "[OK] Python server started" $Green
}
}
if (Test-Path $pyErrLogFile) {
$errContent = Get-Content $pyErrLogFile -Raw -ErrorAction SilentlyContinue
if ($errContent) {
Write-ColorOutput "[DEBUG] Error log: $errContent" $Yellow
}
}
Start-Sleep -Milliseconds 500
}
if (-not $serverStarted) {
Write-ColorOutput "[ERROR] Python server failed to start" $Red
Write-ColorOutput "[DEBUG] Logs:" $Yellow
if (Test-Path $pyLogFile) { Get-Content $pyLogFile | Write-ColorOutput -Color $Yellow }
if (Test-Path $pyErrLogFile) { Get-Content $pyErrLogFile | Write-ColorOutput -Color $Yellow }
if (Test-Path $pyDebugLogFile) { Get-Content $pyDebugLogFile | Write-ColorOutput -Color $Yellow }
Write-ColorOutput "[DEBUG] Trying foreground run..." $Yellow
python -u ping.py server --port 8000
exit 1
}
# Extract Peer ID
$peerInfo = $null
if (Test-Path $pyLogFile) {
$content = Get-Content $pyLogFile -Raw
$peerIdPattern = "Peer ID:\s*([A-Za-z0-9]+)"
$peerIdMatch = [regex]::Match($content, $peerIdPattern)
if ($peerIdMatch.Success) {
$peerId = $peerIdMatch.Groups[1].Value
$peerInfo = @{
PeerId = $peerId
MultiAddr = "/ip4/127.0.0.1/tcp/8000/p2p/$peerId"
}
Write-ColorOutput "[OK] Peer ID: $peerId" $Cyan
Write-ColorOutput "[OK] MultiAddr: $($peerInfo.MultiAddr)" $Cyan
}
}
if (-not $peerInfo) {
Write-ColorOutput "[ERROR] Could not extract Peer ID" $Red
if (Test-Path $pyLogFile) { Get-Content $pyLogFile | Write-ColorOutput -Color $Yellow }
if (Test-Path $pyErrLogFile) { Get-Content $pyErrLogFile | Write-ColorOutput -Color $Yellow }
if (Test-Path $pyDebugLogFile) { Get-Content $pyDebugLogFile | Write-ColorOutput -Color $Yellow }
Stop-Process -Id $pyProcess.Id -Force -ErrorAction SilentlyContinue
exit 1
}
# Start JavaScript client
Write-ColorOutput "[JAVASCRIPT] Starting client..." $Yellow
Set-Location -Path "../js_node"
$jsLogFile = "test_js_client_to_py_server.log"
$jsErrLogFile = "test_js_client_to_py_server.log.err"
if (Test-Path $jsLogFile) { Remove-Item $jsLogFile -Force -ErrorAction SilentlyContinue }
if (Test-Path $jsErrLogFile) { Remove-Item $jsErrLogFile -Force -ErrorAction SilentlyContinue }
$jsProcess = Start-Process -FilePath "node" -ArgumentList "src/ping.js", "client", $peerInfo.MultiAddr, "3" -NoNewWindow -PassThru -RedirectStandardOutput $jsLogFile -RedirectStandardError $jsErrLogFile
Write-ColorOutput "[DEBUG] JavaScript client PID: $($jsProcess.Id)" $Blue
Write-ColorOutput "[DEBUG] Client logs: $((Get-Location).Path)\$jsLogFile, $((Get-Location).Path)\$jsErrLogFile" $Blue
# Wait for client to complete
$clientTimeout = 10
$clientStart = Get-Date
while (-not $jsProcess.HasExited -and (((Get-Date) - $clientStart).TotalSeconds -lt $clientTimeout)) {
Start-Sleep -Seconds 1
}
if (-not $jsProcess.HasExited) {
Write-ColorOutput "[DEBUG] JavaScript client did not exit, terminating..." $Yellow
Stop-Process -Id $jsProcess.Id -Force -ErrorAction SilentlyContinue
}
Write-ColorOutput "[CHECK] Results..." $Cyan
$success = $false
if (Test-Path $jsLogFile) {
$jsLogContent = Get-Content $jsLogFile -Raw -ErrorAction SilentlyContinue
if ($jsLogContent -match "successful|Ping.*successful") {
$success = $true
Write-ColorOutput "[SUCCESS] Ping test passed" $Green
} else {
Write-ColorOutput "[FAILED] No successful pings" $Red
Write-ColorOutput "[DEBUG] Client log path: $((Get-Location).Path)\$jsLogFile" $Yellow
Write-ColorOutput "Client log:" $Yellow
Write-ColorOutput $jsLogContent $Yellow
if (Test-Path $jsErrLogFile) {
Write-ColorOutput "[DEBUG] Client error log path: $((Get-Location).Path)\$jsErrLogFile" $Yellow
Write-ColorOutput "Client error log:" $Yellow
Get-Content $jsErrLogFile | Write-ColorOutput -Color $Yellow
}
Write-ColorOutput "[DEBUG] Python server log path: $((Get-Location).Path)\..\py_node\$pyLogFile" $Yellow
Write-ColorOutput "Python server log:" $Yellow
if (Test-Path "../py_node/$pyLogFile") {
$pyLogContent = Get-Content "../py_node/$pyLogFile" -Raw -ErrorAction SilentlyContinue
if ($pyLogContent) { Write-ColorOutput $pyLogContent $Yellow } else { Write-ColorOutput "Empty or inaccessible" $Yellow }
} else {
Write-ColorOutput "File not found" $Yellow
}
Write-ColorOutput "[DEBUG] Python server error log path: $((Get-Location).Path)\..\py_node\$pyErrLogFile" $Yellow
Write-ColorOutput "Python server error log:" $Yellow
if (Test-Path "../py_node/$pyErrLogFile") {
$pyErrLogContent = Get-Content "../py_node/$pyErrLogFile" -Raw -ErrorAction SilentlyContinue
if ($pyErrLogContent) { Write-ColorOutput $pyErrLogContent $Yellow } else { Write-ColorOutput "Empty or inaccessible" $Yellow }
} else {
Write-ColorOutput "File not found" $Yellow
}
Write-ColorOutput "[DEBUG] Python debug log path: $((Get-Location).Path)\..\py_node\$pyDebugLogFile" $Yellow
Write-ColorOutput "Python debug log:" $Yellow
if (Test-Path "../py_node/$pyDebugLogFile") {
$pyDebugLogContent = Get-Content "../py_node/$pyDebugLogFile" -Raw -ErrorAction SilentlyContinue
if ($pyDebugLogContent) { Write-ColorOutput $pyDebugLogContent $Yellow } else { Write-ColorOutput "Empty or inaccessible" $Yellow }
} else {
Write-ColorOutput "File not found" $Yellow
}
}
}
Write-ColorOutput "[CLEANUP] Stopping processes..." $Yellow
Stop-Process -Id $pyProcess.Id -Force -ErrorAction SilentlyContinue
Stop-Process -Id $jsProcess.Id -Force -ErrorAction SilentlyContinue
Set-Location -Path "../"
if ($success) {
Write-ColorOutput "[SUCCESS] Test completed" $Green
exit 0
} else {
Write-ColorOutput "[FAILED] Test failed" $Red
exit 1
}

View File

@ -1,215 +0,0 @@
#!/usr/bin/env bash
# run_test.sh - libp2p Interoperability Test Runner (Bash)
# Tests py-libp2p <-> js-libp2p ping communication
set -e
# Colors for output
RED='\033[31m'
GREEN='\033[32m'
YELLOW='\033[33m'
BLUE='\033[34m'
CYAN='\033[36m'
RESET='\033[0m'
write_color_output() {
local message="$1"
local color="${2:-$RESET}"
echo -e "${color}${message}${RESET}"
}
write_color_output "[CHECK] Checking prerequisites..." "$CYAN"
if ! command -v python3 &> /dev/null && ! command -v python &> /dev/null; then
write_color_output "[ERROR] Python not found. Install Python 3.7+" "$RED"
exit 1
fi
# Use python3 if available, otherwise python
PYTHON_CMD="python3"
if ! command -v python3 &> /dev/null; then
PYTHON_CMD="python"
fi
if ! command -v node &> /dev/null; then
write_color_output "[ERROR] Node.js not found. Install Node.js 16+" "$RED"
exit 1
fi
write_color_output "[CHECK] Checking port 8000..." "$BLUE"
if netstat -tuln 2>/dev/null | grep -q ":8000 " || ss -tuln 2>/dev/null | grep -q ":8000 "; then
write_color_output "[ERROR] Port 8000 in use. Free the port." "$RED"
if command -v netstat &> /dev/null; then
netstat -tuln | grep ":8000 " | write_color_output "$(cat)" "$YELLOW"
elif command -v ss &> /dev/null; then
ss -tuln | grep ":8000 " | write_color_output "$(cat)" "$YELLOW"
fi
exit 1
fi
write_color_output "[DEBUG] Cleaning up Python processes..." "$BLUE"
pkill -f "ping.py" 2>/dev/null || true
write_color_output "[PYTHON] Starting server on port 8000..." "$YELLOW"
cd py_node
PY_LOG_FILE="py_server_8000.log"
PY_ERR_LOG_FILE="py_server_8000.log.err"
PY_DEBUG_LOG_FILE="ping_debug.log"
rm -f "$PY_LOG_FILE" "$PY_ERR_LOG_FILE" "$PY_DEBUG_LOG_FILE"
$PYTHON_CMD -u ping.py server --port 8000 > "$PY_LOG_FILE" 2> "$PY_ERR_LOG_FILE" &
PY_PROCESS_PID=$!
write_color_output "[DEBUG] Python server PID: $PY_PROCESS_PID" "$BLUE"
write_color_output "[DEBUG] Python logs: $(pwd)/$PY_LOG_FILE, $(pwd)/$PY_ERR_LOG_FILE, $(pwd)/$PY_DEBUG_LOG_FILE" "$BLUE"
TIMEOUT_SECONDS=20
START_TIME=$(date +%s)
SERVER_STARTED=false
while [ $(($(date +%s) - START_TIME)) -lt $TIMEOUT_SECONDS ] && [ "$SERVER_STARTED" = false ]; do
if [ -f "$PY_LOG_FILE" ]; then
if grep -q "Server started\|Listening" "$PY_LOG_FILE" 2>/dev/null; then
SERVER_STARTED=true
write_color_output "[OK] Python server started" "$GREEN"
fi
fi
if [ -f "$PY_ERR_LOG_FILE" ] && [ -s "$PY_ERR_LOG_FILE" ]; then
ERR_CONTENT=$(cat "$PY_ERR_LOG_FILE" 2>/dev/null || true)
if [ -n "$ERR_CONTENT" ]; then
write_color_output "[DEBUG] Error log: $ERR_CONTENT" "$YELLOW"
fi
fi
sleep 0.5
done
if [ "$SERVER_STARTED" = false ]; then
write_color_output "[ERROR] Python server failed to start" "$RED"
write_color_output "[DEBUG] Logs:" "$YELLOW"
[ -f "$PY_LOG_FILE" ] && cat "$PY_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
[ -f "$PY_ERR_LOG_FILE" ] && cat "$PY_ERR_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
[ -f "$PY_DEBUG_LOG_FILE" ] && cat "$PY_DEBUG_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
write_color_output "[DEBUG] Trying foreground run..." "$YELLOW"
$PYTHON_CMD -u ping.py server --port 8000
exit 1
fi
# Extract Peer ID
PEER_ID=""
MULTI_ADDR=""
if [ -f "$PY_LOG_FILE" ]; then
CONTENT=$(cat "$PY_LOG_FILE" 2>/dev/null || true)
PEER_ID=$(echo "$CONTENT" | grep -oP "Peer ID:\s*\K[A-Za-z0-9]+" || true)
if [ -n "$PEER_ID" ]; then
MULTI_ADDR="/ip4/127.0.0.1/tcp/8000/p2p/$PEER_ID"
write_color_output "[OK] Peer ID: $PEER_ID" "$CYAN"
write_color_output "[OK] MultiAddr: $MULTI_ADDR" "$CYAN"
fi
fi
if [ -z "$PEER_ID" ]; then
write_color_output "[ERROR] Could not extract Peer ID" "$RED"
[ -f "$PY_LOG_FILE" ] && cat "$PY_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
[ -f "$PY_ERR_LOG_FILE" ] && cat "$PY_ERR_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
[ -f "$PY_DEBUG_LOG_FILE" ] && cat "$PY_DEBUG_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
kill $PY_PROCESS_PID 2>/dev/null || true
exit 1
fi
# Start JavaScript client
write_color_output "[JAVASCRIPT] Starting client..." "$YELLOW"
cd ../js_node
JS_LOG_FILE="test_js_client_to_py_server.log"
JS_ERR_LOG_FILE="test_js_client_to_py_server.log.err"
rm -f "$JS_LOG_FILE" "$JS_ERR_LOG_FILE"
node src/ping.js client "$MULTI_ADDR" 3 > "$JS_LOG_FILE" 2> "$JS_ERR_LOG_FILE" &
JS_PROCESS_PID=$!
write_color_output "[DEBUG] JavaScript client PID: $JS_PROCESS_PID" "$BLUE"
write_color_output "[DEBUG] Client logs: $(pwd)/$JS_LOG_FILE, $(pwd)/$JS_ERR_LOG_FILE" "$BLUE"
# Wait for client to complete
CLIENT_TIMEOUT=10
CLIENT_START=$(date +%s)
while kill -0 $JS_PROCESS_PID 2>/dev/null && [ $(($(date +%s) - CLIENT_START)) -lt $CLIENT_TIMEOUT ]; do
sleep 1
done
if kill -0 $JS_PROCESS_PID 2>/dev/null; then
write_color_output "[DEBUG] JavaScript client did not exit, terminating..." "$YELLOW"
kill $JS_PROCESS_PID 2>/dev/null || true
fi
write_color_output "[CHECK] Results..." "$CYAN"
SUCCESS=false
if [ -f "$JS_LOG_FILE" ]; then
JS_LOG_CONTENT=$(cat "$JS_LOG_FILE" 2>/dev/null || true)
if echo "$JS_LOG_CONTENT" | grep -q "successful\|Ping.*successful"; then
SUCCESS=true
write_color_output "[SUCCESS] Ping test passed" "$GREEN"
else
write_color_output "[FAILED] No successful pings" "$RED"
write_color_output "[DEBUG] Client log path: $(pwd)/$JS_LOG_FILE" "$YELLOW"
write_color_output "Client log:" "$YELLOW"
write_color_output "$JS_LOG_CONTENT" "$YELLOW"
if [ -f "$JS_ERR_LOG_FILE" ]; then
write_color_output "[DEBUG] Client error log path: $(pwd)/$JS_ERR_LOG_FILE" "$YELLOW"
write_color_output "Client error log:" "$YELLOW"
cat "$JS_ERR_LOG_FILE" | while read line; do write_color_output "$line" "$YELLOW"; done
fi
write_color_output "[DEBUG] Python server log path: $(pwd)/../py_node/$PY_LOG_FILE" "$YELLOW"
write_color_output "Python server log:" "$YELLOW"
if [ -f "../py_node/$PY_LOG_FILE" ]; then
PY_LOG_CONTENT=$(cat "../py_node/$PY_LOG_FILE" 2>/dev/null || true)
if [ -n "$PY_LOG_CONTENT" ]; then
write_color_output "$PY_LOG_CONTENT" "$YELLOW"
else
write_color_output "Empty or inaccessible" "$YELLOW"
fi
else
write_color_output "File not found" "$YELLOW"
fi
write_color_output "[DEBUG] Python server error log path: $(pwd)/../py_node/$PY_ERR_LOG_FILE" "$YELLOW"
write_color_output "Python server error log:" "$YELLOW"
if [ -f "../py_node/$PY_ERR_LOG_FILE" ]; then
PY_ERR_LOG_CONTENT=$(cat "../py_node/$PY_ERR_LOG_FILE" 2>/dev/null || true)
if [ -n "$PY_ERR_LOG_CONTENT" ]; then
write_color_output "$PY_ERR_LOG_CONTENT" "$YELLOW"
else
write_color_output "Empty or inaccessible" "$YELLOW"
fi
else
write_color_output "File not found" "$YELLOW"
fi
write_color_output "[DEBUG] Python debug log path: $(pwd)/../py_node/$PY_DEBUG_LOG_FILE" "$YELLOW"
write_color_output "Python debug log:" "$YELLOW"
if [ -f "../py_node/$PY_DEBUG_LOG_FILE" ]; then
PY_DEBUG_LOG_CONTENT=$(cat "../py_node/$PY_DEBUG_LOG_FILE" 2>/dev/null || true)
if [ -n "$PY_DEBUG_LOG_CONTENT" ]; then
write_color_output "$PY_DEBUG_LOG_CONTENT" "$YELLOW"
else
write_color_output "Empty or inaccessible" "$YELLOW"
fi
else
write_color_output "File not found" "$YELLOW"
fi
fi
fi
write_color_output "[CLEANUP] Stopping processes..." "$YELLOW"
kill $PY_PROCESS_PID 2>/dev/null || true
kill $JS_PROCESS_PID 2>/dev/null || true
cd ../
if [ "$SUCCESS" = true ]; then
write_color_output "[SUCCESS] Test completed" "$GREEN"
exit 0
else
write_color_output "[FAILED] Test failed" "$RED"
exit 1
fi

View File

@ -0,0 +1,5 @@
def test_js_libp2p_placeholder():
"""
Placeholder test for js-libp2p interop tests.
"""
assert True, "Placeholder test for js-libp2p interop tests"

View File

@ -24,22 +24,16 @@ def make_pubsub_msg(
)
# TODO: Implement sparse connect
async def dense_connect(hosts: Sequence[IHost]) -> None:
await connect_some(hosts, 10)
# FIXME: `degree` is not used at all
async def connect_some(hosts: Sequence[IHost], degree: int) -> None:
"""
Connect each host to up to 'degree' number of other hosts.
Creates a sparse network topology where each node has limited connections.
"""
for i, host in enumerate(hosts):
connections_made = 0
for j in range(i + 1, len(hosts)):
if connections_made >= degree:
break
await connect(host, hosts[j])
connections_made += 1
for host2 in hosts[i + 1 :]:
await connect(host, host2)
async def one_to_all_connect(hosts: Sequence[IHost], central_host_index: int) -> None:

View File

@ -1,31 +0,0 @@
from unittest.mock import (
MagicMock,
)
import trio
from libp2p.abc import IHost
def create_mock_connections(count: int = 50) -> dict:
connections = {}
for i in range(1, count):
peer_id = f"peer-{i}"
mock_conn = MagicMock(name=f"INetConn-{i}")
connections[peer_id] = mock_conn
return connections
async def run_host_forever(host: IHost, addr):
async with host.run([addr]):
await trio.sleep_forever()
async def wait_until_listening(host, timeout=3):
with trio.move_on_after(timeout):
while not host.get_addrs():
await trio.sleep(0.05)
return
raise RuntimeError("Timed out waiting for host to get an address")