So the main thing is just the transfer speeds at the host, correct?
Nope. It does matters how many resources are free at the time.
data transit speed and connectivity only take care of download of backup from one server to another.
backup restoration do require little resources. time differs case by case basis.
1) manual : you will create full / partial backup and download > upload to new server file manager > restore backup by yourself or raise ticket if wish for restoration done by hosting provider.
This is time consuming process and required time depends on responsiveness of support staff(If want it to done by provider). [provider will use only backup restoration tools]
2) Auto (shared hosting) : In this process you create full backup and keep in old hosting provider > raise ticket at new hosting provider requesting full backup restoration. you will require to provide your cpanel login details of old hosting account. [provide will use transfer tools]
3) Auto (VPS) : If you have whm panel access on both hosting provider, Yes. this can be done within few minutes by using transfer tools by yourself.
Timeframe : case 1, 2 : An hour minimum
case 3 : Few minutes