Files
py-libp2p/Makefile
Sumanjeet d61bca78ab Kademlia DHT implementation in py-libp2p (#579)
* initialise the module

* added content routing

* added routing module

* added peer routing

* added value store

* added utilities functions

* added main kademlia file

* fixed create_key_from_binary function

* example to test kademlia dht

* added protocol ID and enhanced logging for peer store size in provider and consumer nodes

* refactor: specify stream type in handle_stream method and add peer in routing table

* removed content routing

* added default value of count for finding closest peers

* added functions to find close peers

* refactor: remove content routing and enhance peer discovery

* added put value function

* added get value function

* fix: improve logging and handle key encoding in get_value method

* refactor: remove ContentRouting import from __init__.py

* refactor: improved basic kademlia example

* added protobuf files

* replaced json with protobuf

* refactor: enhance peer discovery and routing logic in KadDHT

* refactor: enhance Kademlia routing table to use PeerInfo objects and improve peer management

* refactor: enhance peer addition logic to utilize PeerInfo objects in routing table

* feat: implement content provider functionality in Kademlia DHT

* refactor: update value store to use datetime for validity management

* refactor: update RoutingTable initialization to include host reference

* refactor: enhance KBucket and RoutingTable for improved peer management and functionality

* refactor: streamline peer discovery and value storage methods in KadDHT

* refactor: update KadDHT and related classes for async peer management and enhanced value storage

* refactor: enhance ProviderStore initialization and improve peer routing integration

* test: add tests for Kademlia DHT functionality

* fix linting issues

* pydocstyle issues fixed

* CICD pipeline issues solved

* fix: update docstring format for find_peer method

* refactor: improve logging and remove unused code in DHT implementation

* refactor: clean up logging and remove unused imports in DHT and test files

* Refactor logging setup and improve DHT stream handling with varint length prefixes

* Update bootstrap peer handling in basic_dht example and refactor peer routing to accept string addresses

* Enhance peer querying in Kademlia DHT by implementing parallel queries using Trio.

* Enhance peer querying by adding deduplication checks

* Refactor DHT implementation to use varint for length prefixes and enhance logging for better traceability

* Add base58 encoding for value storage and enhance logging in basic_dht example

* Refactor Kademlia DHT to support server/client modes

* Added unit tests

* Refactor documentation to fixsome warning

* Add unit tests and remove outdated tests

* Fixed precommit errora

* Refactor error handling test to raise StringParseError for invalid bootstrap addresses

* Add libp2p.kad_dht to the list of subpackages in documentation

* Fix expiration and republish checks to use inclusive comparison

* Add __init__.py file to libp2p.kad_dht.pb package

* Refactor get value and put value to run in parallel with query timeout

* Refactor provider message handling to use parallel processing with timeout

* Add methods for provider store in KadDHT class

* Refactor KadDHT and ProviderStore methods to improve type hints and enhance parallel processing

* Add documentation for libp2p.kad_dht.pb module.

* Update documentation for libp2p.kad_dht package to include subpackages and correct formatting

* Fix formatting in documentation for libp2p.kad_dht package by correcting the subpackage reference

* Fix header formatting in libp2p.kad_dht.pb documentation

* Change log level from info to debug for various logging statements.

* fix CICD issues (post revamp)

* fixed value store unit test

* Refactored kademlia example

* Refactor Kademlia example: enhance logging, improve bootstrap node connection, and streamline server address handling

* removed bootstrap module

* Refactor Kademlia DHT example and core modules: enhance logging, remove unused code, and improve peer handling

* Added docs of kad dht example

* Update server address log file path to use the script's directory

* Refactor: Introduce DHTMode enum for clearer mode management

* moved xor_distance function to utils.py

* Enhance logging in ValueStore and KadDHT: include decoded value in debug logs and update parameter description for validity

* Add handling for closest peers in GET_VALUE response when value is not found

* Handled failure scenario for PUT_VALUE

* Remove kademlia demo from project scripts and contributing documentation

* spelling and logging

---------

Co-authored-by: pacrob <5199899+pacrob@users.noreply.github.com>
2025-06-16 14:46:40 -06:00

156 lines
4.6 KiB
Makefile

CURRENT_SIGN_SETTING := $(shell git config commit.gpgSign)
.PHONY: clean-pyc clean-build docs
help:
@echo "clean-build - remove build artifacts"
@echo "clean-pyc - remove Python file artifacts"
@echo "clean - run clean-build and clean-pyc"
@echo "dist - build package and cat contents of the dist directory"
@echo "fix - fix formatting & linting issues with ruff"
@echo "lint - fix linting issues with pre-commit"
@echo "test - run tests quickly with the default Python"
@echo "docs - generate docs and open in browser (linux-docs for version on linux)"
@echo "package-test - build package and install it in a venv for manual testing"
@echo "notes - consume towncrier newsfragments and update release notes in docs - requires bump to be set"
@echo "release - package and upload a release (does not run notes target) - requires bump to be set"
@echo "pr - run clean, fix, lint, typecheck, and test i.e basically everything you need to do before creating a PR"
clean-build:
rm -fr build/
rm -fr dist/
rm -fr *.egg-info
clean-pyc:
find . -name '*.pyc' -exec rm -f {} +
find . -name '*.pyo' -exec rm -f {} +
find . -name '*~' -exec rm -f {} +
find . -name '__pycache__' -exec rm -rf {} +
clean: clean-build clean-pyc
dist: clean
python -m build
ls -l dist
lint:
@pre-commit run --all-files --show-diff-on-failure || ( \
echo "\n\n\n * pre-commit should have fixed the errors above. Running again to make sure everything is good..." \
&& pre-commit run --all-files --show-diff-on-failure \
)
fix:
python -m ruff check --fix
typecheck:
pre-commit run mypy-local --all-files && pre-commit run pyrefly-local --all-files
test:
python -m pytest tests -n auto
pr: clean fix lint typecheck test
# protobufs management
PB = libp2p/crypto/pb/crypto.proto \
libp2p/pubsub/pb/rpc.proto \
libp2p/security/insecure/pb/plaintext.proto \
libp2p/security/secio/pb/spipe.proto \
libp2p/security/noise/pb/noise.proto \
libp2p/identity/identify/pb/identify.proto \
libp2p/host/autonat/pb/autonat.proto \
libp2p/kad_dht/pb/kademlia.proto
PY = $(PB:.proto=_pb2.py)
PYI = $(PB:.proto=_pb2.pyi)
## Set default to `protobufs`, otherwise `format` is called when typing only `make`
all: protobufs
protobufs: $(PY)
%_pb2.py: %.proto
protoc --python_out=. --mypy_out=. $<
clean-proto:
rm -f $(PY) $(PYI)
# docs commands
docs: check-docs
open docs/_build/html/index.html
linux-docs: check-docs
xdg-open docs/_build/html/index.html
# docs helpers
validate-newsfragments:
python ./newsfragments/validate_files.py
towncrier build --draft --version preview
check-docs: build-docs validate-newsfragments
build-docs:
sphinx-apidoc -o docs/ . "*conftest*" tests/
$(MAKE) -C docs clean
$(MAKE) -C docs html
$(MAKE) -C docs doctest
check-docs-ci: build-docs build-docs-ci validate-newsfragments
build-docs-ci:
$(MAKE) -C docs epub
# release commands
package-test: clean
python -m build
python scripts/release/test_package.py
notes: check-bump validate-newsfragments
# Let UPCOMING_VERSION be the version that is used for the current bump
$(eval UPCOMING_VERSION=$(shell bump-my-version bump --dry-run $(bump) -v | awk -F"'" '/New version will be / {print $$2}'))
# Now generate the release notes to have them included in the release commit
towncrier build --yes --version $(UPCOMING_VERSION)
# Before we bump the version, make sure that the towncrier-generated docs will build
make build-docs
git commit -m "Compile release notes for v$(UPCOMING_VERSION)"
release: check-bump check-git clean
# verify that notes command ran correctly
./newsfragments/validate_files.py is-empty
CURRENT_SIGN_SETTING=$(git config commit.gpgSign)
git config commit.gpgSign true
bump-my-version bump $(bump)
python -m build
git config commit.gpgSign "$(CURRENT_SIGN_SETTING)"
git push upstream && git push upstream --tags
twine upload dist/*
# release helpers
check-bump:
ifndef bump
$(error bump must be set, typically: major, minor, patch, or devnum)
endif
check-git:
# require that upstream is configured for libp2p/py-libp2p
@if ! git remote -v | grep "upstream[[:space:]]git@github.com:libp2p/py-libp2p.git (push)\|upstream[[:space:]]https://github.com/libp2p/py-libp2p (push)"; then \
echo "Error: You must have a remote named 'upstream' that points to 'py-libp2p'"; \
exit 1; \
fi
# autonat specific protobuf targets
format-autonat-proto:
black libp2p/host/autonat/pb/autonat_pb2*.py*
isort libp2p/host/autonat/pb/autonat_pb2*.py*
autonat-proto: clean-autonat
protoc --python_out=. --mypy_out=. libp2p/host/autonat/pb/autonat.proto
$(MAKE) format-autonat-proto
clean-autonat:
rm -f libp2p/host/autonat/pb/autonat_pb2*.py*