Nakamoto: 2.5/3.0 Node should migrate from 2.4.x chainstate

[kantai]

The Nakamoto stacks-node should be able to migrate from 2.4.x chainstates.

[jcnelson]

Just wanted to say, this is a lot more reasonable (and a lot less scary) than what I thought was being suggesting earlier. It had sounded like we were going to do away with genesis sync altogether, which would have been much more difficult if not impossible to work with to recover from SIP-011-type catastrophes like the 2.1 --> 2.4 transition (which I don't think could be safely achieved without a genesis sync).

That said, we made the decision early on in next to treat the 2.5 schema as a breaking change, so it's an open question how much work will be required to implement and test a migration path that we can be proud of.

[kantai]

Just wanted to say, this is a lot more reasonable (and a lot less scary) than what I thought was being suggesting earlier. It had sounded like we were going to do away with genesis sync altogether, which would have been much more difficult if not impossible to work with to recover from SIP-011-type catastrophes like the 2.1 --> 2.4 transition (which I don't think could be safely achieved without a genesis sync).

Oh yeah -- that's not what I am suggesting. The ability to do genesis syncs is very important, I agree. I think it's just much easier on the network to not require them on updates.

That said, we made the decision early on in next to treat the 2.5 schema as a breaking change, so it's an open question how much work will be required to implement and test a migration path that we can be proud of.

Yes -- I agree, this requires some investigation. I think it's possible that its still not that much yet -- NakamotoBlocks are using entirely new tables after all. There are some new columns, etc. elsewhere, but I think that we should be able to enumerate them.

[jcnelson]

I think it's just much easier on the network to not require them on updates.

Yes, 100% agreed. I'd be in favor of making chainstate backwards-compatibility a requirement going forward for all releases, including hard forks. We'll want to spend some time thinking about how best to test this, but I have a few ideas (e.g. we could save a bunch of chainstate snapshots from legacy nodes for just this purpose in order to verify that all previous snapshots migrate to the latest schemas, for example).

[zone117x]

Thanks @cylewitruk, this will save an enormous amount of time across the ecosystem

[cylewitruk]

I'm going to wrap up what I'm currently working on with proptest and then I'll jump on this in a couple of days ^^

@jcnelson @kantai If there are any hints/tips/gotcha's you'd like to drop here before I get started, it might save some time later. Thinking mostly about new columns which may need to be derived in some way -- if you know how they should be derived, etc. Or if new columns can have some default value. I think you get what I'm after.. :)

[ASuciuX]

I tried to run replay-block with the latest next (commit), as I did before on the 99,990-100k blocks, and it crashes. I think something was added for newer blocks to create the signers signature, but the code runs with previous blocks the same way and it throws as that field doesn’t exist.

computer@debian:~/Desktop/profiling/asuciux-core/target/release$ hyperfine -w 3 -r 10 "./stacks-inspect-d3f1a3164-17-mar replay-block ~/chainstate range 99990 100000" --show-output

Benchmark 1: ./stacks-inspect-d3f1a3164-17-mar replay-block ~/chainstate range 99990 100000
Will check 10 blocks
Checked 0...
thread 'main' panicked at stackslib/src/main.rs:1578:6:
called `Result::unwrap()` on an `Err` value: SqliteError(SqliteFailure(Error { code: Unknown, extended_code: 1 }, Some("no such table: main.vote_for_aggregate_key")))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Command terminated with non-zero exit code: 101. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.