This is almost always caused by your login scripts on the server generating output. PSCP or PSFTP will receive that output when they were expecting to see the start of a file transfer protocol, and they will attempt to interpret the output as file-transfer protocol. This will usually lead to an ‘out of memory’ error for much the same reasons as given in question A.7.3.
This is a setup problem in your account on your server, not a PSCP/PSFTP bug. Your login scripts should never generate output during non-interactive sessions; secure file transfer is not the only form of remote access that will break if they do.
On Unix, a simple fix is to ensure that all the parts of your login script that might generate output are in .profile (if you use a Bourne shell derivative) or .login (if you use a C shell). Putting them in more general files such as .bashrc or .cshrc is liable to lead to problems.