I get what your saying. I think movement of people has a relevance. The mistake is to compare inward investment and international trade deals to a Building society loan. I wouldnt expect a Barclays Mortgage employee to take up residence in my house.
Just about every trade deal Ive seen includes an element of free movement (or certainly guarateed easy access visas). Both China and India have said in the last few years Britain will not get our money without our people. The thinking isnt strictly quid pro quo. More that employees need guaranteed quick easy access to the recipient country. A China trade deal with Britain would include easy access for students (that they see as a long term benefit to their country).
China is doing a lot of investment in Africa and,South East Asia where access for their wirkers is a key part of the deal (as China moves to higher value production, Africa and South East Asia will become the next low cost production workshops).
3 of the 4 pillars of the EU are perfectly acceptable to most Brexiteers. Miller Bill will take foreign money but baulk at the idea accepting their people.