Adapter Merging
Weight deltas from separately fine-tuned models can be added, negated and interpolated. Why that works at all is not settled.
This entry is a stub. The maths has not been rederived by hand, or the implementation has not been run on real tensors, so it is recorded here as an open question rather than an answer.
Read what follows as a pointer to the sources, not as a settled account.
Write for the delta a fine-tune produced. Task arithmetic observes that often behaves like a model trained on the union of those tasks, and that negating a subtracts the behaviour. TIES and DARE add conflict resolution: trim small entries, elect a sign per coordinate, and average only the agreeing deltas.
The reason this is a stub is that the explanation is not settled. Linear mode connectivity between fine-tunes of a shared base is the usual account, but it is a description of when merging works rather than a prediction, and the failure cases have not been rederived here.