Hi,
You're correct, you can't just have the value of MSH4 as the reference.
As it looks to me, there are several options. If you look at the mapping samples here (warning - not an official source of truth in any way), you will see that some mappers (Philips) just use the fields source/name and source/endpoint for MSH3 and MSH4 (instead of sender), and some (Google) actually add a new Organization resource to the message bundle and then directly reference it in the header. The last one seems problematic to me, since it will add more and more Organization resources with each message, unless some conditional trickery is applied.
However, there is another way. The Reference type can have an 'identifier' filed instead of a 'reference' field in cases there is no way to directly reference an existing resource (look here). In this case, you just set the value of MSH4 to the sender/identifier/@value and your custom codesystem to sender/identifier/@system, and adhere to the guidelines. It doesn't have to be a reference to an actual existing resource in the FHIR server, but it will be conveniently reachable and searchable.