Depend on `serde_core` instead of `serde`

[SludgePhD]

We don't use the proc macros (I think?), so this can save some compile time.