Commit Graph

16 Commits

Author SHA1 Message Date
Bo QIU
82fef674ed
Add more metrics for file sync ()
* add metrics in log sync package

* udpate auto sync metrics

* Add metrics for completed file sync

* add more metrics for serial file sync

* adjust default timeout value for auto sync

* fix metrics rpc for Timer type

* add metrics for channel

* refactor channel metrics

* add timeout metrics for segment sync

* refactor channel receiver
2024-08-22 10:42:15 +08:00
Bo QIU
e1df2f0868
Change hardcoded const variables to configs () 2024-08-08 16:21:50 +08:00
Bo QIU
ae9c52c0e6
Terminate file sync if connecting peers too long () 2024-07-27 21:05:44 +08:00
peilun-conflux
ba8d065e73
Generate sync proof with the given tx seq. ()
* Generate sync proof with the given tx seq.

* Fill HistoryTree with new data.

* Fix clippy.
2024-07-12 15:55:57 +08:00
Bo QIU
2bc402f94b
Enhance sync logs ()
* enhance sync logs

* fix lint
2024-07-12 14:26:23 +08:00
Bo QIU
84f455d1d4
Fix some issues in sync package ()
* Change default concurrent sync threads

* enhance log level in sync store

* Add rwlock in sync store to avoid concurrency issue

* enhance serial sync state machine log level

* wrap Instant for debug purpose

* add timestamp for connecting peers state

* Select random peer to sync chunks

* return sync peers states instead of connected peers only

* Use tokio async RwLock for sync store
2024-07-08 11:47:59 +08:00
Bo QIU
055d1fa32d
Fix sync state machine issue for sharding case ()
* Fix sync state machine issue for sharding case

* Fix fmt lint
2024-07-01 16:35:27 +08:00
peilun-conflux
4eb2a50b0e
Use inner lock in storage and use async lock. ()
* Use inner lock in storage.

* Remove mut.

* Remove async lock for storage.

* Fix tests and warnings.

* Use spawn_blocking for storage task.

* Fix clippy.

* Finalize the new tx at last.

* Revert "Finalize the new tx at last."

This reverts commit b56ad5582d.

* Wait for old same-root txs to finalize.

* Use async storage in miner.

* Update rust version to 1.79.0.

* Use Vec to avoid stack overflow.

* Fix unused warning.

* Fix clippy.

* Fix test warning.

* Fix test.

* fmt.

* Use async storage in pruner.

* nit.
2024-06-29 17:08:02 +08:00
Bo QIU
439314372b
Remove file location cache for connection timeout peer () 2024-06-27 01:51:37 +08:00
Bo QIU
c6325f7643
Fix file sync pending due to peer connection timeout ()
* enhance logs for sync module

* report peer if connection timeout

* Use origin timestamp of sync state to determine if sync too long to terminate

* fix compilation error
2024-06-25 17:47:25 +08:00
peilun-conflux
0f52325f67
Sync and finalize sharded data. ()
* Save test logs.

* Prepare to handle shard config in chunk pool.

* Allow files to be finalized with only sharded data.

* Handle shard config change.

* Only sync needed data in shard config.

* Fix clippy and tests.
2024-06-08 02:50:36 +08:00
peilun-conflux
c2c6e2d5fb
Store shard config for peers and choose sync peers accordingly. ()
* Implement Pruner.

* Put pruner in a crate.

* Fix clippy.

* Add rpc zgs_getShardConfig.

* Fix.

* Increase wait time.

* Add pruner_test and use max_num_chunks instead of size_limit.

* Store shard config for peers and choose sync peers accordingly.

* Add test and fix sync.

* Fix clippy and test.

* Fix some ut.

* Add AnnounceShardConfig gossip and fix tests.

* Add sharded tx finalize check in LogManager.

* Try,

* Rename.

* Longer timeout for mine_test.

* Save test logs.
2024-06-07 16:58:15 +08:00
Bo QIU
c2b8b1cab3
Rpc enhancements ()
* add admin rpc to terminate file or chunks sync

* Disallow to change file sync goal when failed

* split admin rpc into separate endpoints

* Support to launch public and private rpc at one port

* use one rpc port for python test

* fix lint

* restart to sync file or chunks if sync already completed or failed
2024-01-30 16:50:35 +08:00
peilun-conflux
0c123500c9
Update the merkle tree with proof data. ()
* Add proof data for chunk proof.

* Support file proof.

* Update with segment proof and fix issues.

* Fix more issues.

* Fix the process of file proof.

* Merge branch 'main' into fix_proof

* Enable sync_test.

* Fix wrongly updated submodule.

* Fix bsc node version.
2024-01-26 10:29:09 +08:00
Bo QIU
9b4b0436c3
Supports to sync partial chunks ()
* refactor p2p signed message

* add new pubsub messages in network layer to find chunks

* handle find chunks pubsub message in router

* Supports to sync partial chunks

* add admin rpc to sync chunks

* limit number of chunks to sync at a time

* refactor code to sync file and chunks

* add more switches to trigger file sync

* fix ut failure

* refactor code
2024-01-19 14:04:59 +08:00
Peilun Li
9058467be1 Initial commit 2024-01-04 18:05:32 +08:00