Because the receiver address is required to uniquely identify a given flow.

Consider the case where two different clients connect to the same web server, and they happen to pick the same ephemeral port for their outbound connection. The flow from those two clients to the web server is uniquely identified in this case only by the sender address. But web servers send data back, and when that happens, the sender/receiver roles are reversed. If you don’t include the receiver address as part of the multiplexing key, you cannot uniquely identify the return traffic in such a case.

That sounds like a possibly rare situation, but there are only 16383 ephemeral ports. That means you’re obviously guaranteed to get overlap if the node is serving more than that many clients simultaneously (which is absolutely reasonable for a large site), but it’s actually worse than that. Assuming that each host picks an ephemeral port randomly with a uniform probability distribution, you only need 152 clients to get a 50% chance of two picking the same port, and 389 for a 99% chance (this is a generalized case of the Birthday Problem).

Also, because the addresses involved are always inherently part of the individual packets anyway, not using them in the multiplexing key doesn’t actually hide anything from nodes that those packets are traversing, which means it’s not really any use in trying to defeat censorship or information harvesting.