Merge pull request #1142 from FactomProject/develop

End of Factom Era

.gitignore

@@ -28,7 +28,6 @@ _obj
 _test
 build
 bin
-pkg
 .vagrant
 
 # Architecture specific extensions/prefixes

Utilities/snapshot/.gitignore

@@ -0,0 +1 @@
+snapshot

Utilities/snapshot/README.md

@@ -0,0 +1,72 @@
+# Snapshot CLI Tool
+
+The snapshot CLI tool is used to download all factoid balances and entry data into a easy to parse format. The tool will read from a factom database, defaulting to the main database path `$HOME/.factom/m2/main-database/ldb/MAIN/factoid_level.db`. **By default entry data is not snapshotted**. Use `-e true` to include entry data. 
+
+All output will be written to a directory, default `./snapshot`.
+
+## CLI
+
+Use `snapshot new` to take a new snapshot of a given database. To delete the snapshot, you can use `snapshot clean`.
+
+You can run against an API of a running node:
+```
+snapshot new --db-type=api --db=http://localhost:8088
+```
+
+```bash
+Usage:
+  snapshot [flags]
+  snapshot [command]
+
+Available Commands:
+  clean       Deletes the snapshotted data.
+  completion  Generate the autocompletion script for the specified shell
+  help        Help about any command
+  new         Take a new snapshot of a factom database
+  verify      verifies the snapshotted data against factom
+
+Flags:
+  -h, --help         help for snapshot
+      --log string   set the log level (default "debug")
+
+Use "snapshot [command] --help" for more information about a command.
+
+```
+
+# Data Formats
+
+## Balances
+
+The `balances` file will have `height <block_height>` for the height the snapshot was taken at. The `block_height`'s factoid block is included in the snapshot. All factoid balances are in factoshis.
+
+```
+height 246710
+FA2FKpXhyWPGpQE9yQ7dc419n4eBW9NrPGcFXhuUQYn8BJ6wYU6H: 0
+FA1zS79XhiyLRrvGGE6aADACt9Uh5oiBoyf9nYgN22HwGkYcAokP: 87975143
+EC2nGxgm8LMaTY6zTE2xzvkoba61hkHy5Uxmn8Zc3udoqwuampaF: 450
+EC3FzWegxBoX7Hk7ZHvSjpcpKeeN1KDPscwBf2C1ejmybSWMQJCr: 42
+```
+
+## Entries
+
+Entries will be written to a `snapshot/entries` directory in flat files for each chain. Each chain file will have its entries written in order with a newline for each entry. Entry lines are prefixed with `et:` followed by the marshaled entry. Eblocks are prefixed with `eb:` and contain their height and keymr. Minute markers are omitted.
+
+```
+eb:4340 176d5e0e0cecfce6887268cb5753615fd10b3011f34c2654a5c8cdad9eb08a19
+et:<entry_binary in base64>
+```
+
+# Verify snapshot
+
+You can verify the snapshot against a running node. Keep in mind, if a node is on a running network, then the balances will mismatch. Ideally you check against a node that loaded a db and is not syncing any network.
+
+```
+snapshot verify balances
+snapshot verify chains
+```
+
+# TODO
+
+It could be optimized with go routines, and there might be a file limit for chains. So I might need to rotate the file cache and only have N number of file descriptors open. Eg use something like https://github.com/hashicorp/golang-lru
+
+Currently `verify chains` does not prove **all** chains were recorded. We can do this by tallying up the chains in the system, and compare to the number of chains we recorded.

Utilities/snapshot/go.mod

@@ -0,0 +1,65 @@
+module github.com/FactomProject/factomd/Utilities/snapshot
+
+go 1.17
+
+require (
+	github.com/AlecAivazis/survey/v2 v2.3.2
+	github.com/FactomProject/FactomCode v0.3.6-0.20171228170625-d7e03150a9d5
+	github.com/FactomProject/factom v0.3.6-0.20190712163801-e7717c4ab072
+	github.com/FactomProject/factomd v1.13.0
+	github.com/sirupsen/logrus v1.8.1
+	github.com/spf13/cobra v1.4.0
+	github.com/stretchr/testify v1.5.1
+)
+
+// Use the local factomd
+replace github.com/FactomProject/factomd => ../../
+
+require (
+	github.com/FactomProject/basen v0.0.0-20150613233007-fe3947df716e // indirect
+	github.com/FactomProject/bolt v1.1.1-0.20170202195217-952a1b4e9a55 // indirect
+	github.com/FactomProject/btcutil v0.0.0-20200312214114-5fd3eaf71bd2 // indirect
+	github.com/FactomProject/btcutilecc v0.0.0-20130527213604-d3a63a5752ec // indirect
+	github.com/FactomProject/dynrsrc v0.3.1 // indirect
+	github.com/FactomProject/ed25519 v0.0.0-20150814230546-38002c4fe7b6 // indirect
+	github.com/FactomProject/factoid v0.3.4 // indirect
+	github.com/FactomProject/fsnotify v0.9.0 // indirect
+	github.com/FactomProject/go-bip32 v0.3.6-0.20161206200006-3b593af1c415 // indirect
+	github.com/FactomProject/go-bip39 v0.3.6-0.20161217174232-d1007fb78d9a // indirect
+	github.com/FactomProject/go-bip44 v0.0.0-20190306062959-b541a96d8da9 // indirect
+	github.com/FactomProject/go-simplejson v0.5.0 // indirect
+	github.com/FactomProject/go-spew v0.0.0-20160301052117-ddfaec9b42f5 // indirect
+	github.com/FactomProject/gocoding v0.0.0-20150814232539-59666ce39524 // indirect
+	github.com/FactomProject/goleveldb v0.2.2-0.20170418171130-e7800c6976c5 // indirect
+	github.com/FactomProject/logrustash v0.0.0-20171005151533-9c7278ede46e // indirect
+	github.com/FactomProject/netki-go-partner-client v0.0.0-20160324224126-426acb535e66 // indirect
+	github.com/FactomProject/snappy-go v0.0.0-20170202213131-f2f83b22c29e // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/boltdb/bolt v1.3.1 // indirect
+	github.com/btcsuitereleases/btcutil v0.0.0-20150612230727-f2b1058a8255 // indirect
+	github.com/cespare/xxhash/v2 v2.1.1 // indirect
+	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
+	github.com/gogo/protobuf v1.3.1-0.20190908201246-8a5ed79f6888 // indirect
+	github.com/golang/protobuf v1.3.4 // indirect
+	github.com/gorilla/mux v1.7.4-0.20190720201435-e67b3c02c719 // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
+	github.com/mattn/go-colorable v0.1.2 // indirect
+	github.com/mattn/go-isatty v0.0.8 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
+	github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect
+	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/prometheus/client_golang v1.5.1 // indirect
+	github.com/prometheus/client_model v0.2.0 // indirect
+	github.com/prometheus/common v0.9.1 // indirect
+	github.com/prometheus/procfs v0.0.8 // indirect
+	github.com/rs/cors v1.7.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	golang.org/x/crypto v0.0.0-20200311171314-f7b00557c8c4 // indirect
+	golang.org/x/sys v0.0.0-20201119102817-f84b799fce68 // indirect
+	golang.org/x/term v0.0.0-20210503060354-a79de5458b56 // indirect
+	golang.org/x/text v0.3.3 // indirect
+	gopkg.in/gcfg.v1 v1.2.3 // indirect
+	gopkg.in/warnings.v0 v0.1.2 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+)