First of all the address parts are in the IP layer, not the TCP/UDP/Transport layer itself. But answering the question further regardless.
I would list all purposes that come to mind:
- you need the receiver address in the packet so that routers know where to forward your packet, otherwise they wouldn't know whether packet is for them as a device or to be forwarded
- receiver itself needs to know whether the packet is for them. In some networks, nodes can see packets for other nodes. For example when using hubs or if improperly speced switch is used with too many connected nodes
- receiver should know which of the local addresses needs to be used. Locally you may use different services on the different IPs
- there are also multicast and broadcast addresses that usually use UDP for transport. Obviously routers and receivers need to know what kind of address the packet is destined for
- useful for firewalls
Fragmenting IP diagrams into diagrams with various IP addresses of the same receiver could at least in theory help Internet freedom.
Where would you put all these "various IP addresses"?