How malware triggers without user running an executable?
It is known that downloading a file without actually executing it can be a security vulnerability, if the operating system does something which is not known by the user. See here for some good answers.
However, I could not find a detailed explanation on how some of those things are done. Suppose that after downloading a file, the OS triggers some process to scan the files, for "antivirus" or "thumbnails", and the file utilises the vulnerability of the process scanning it to execute arbitrary code. My question is, how can arbitrary code which has never been written into the vulnerable scanner process suddenly be injected into it?
Let's assume the buggy process is written in C. For most C programs, all memory allocated are not executable, so injecting code into it does not have any security issues. Therefore, only a small collection of C programs which implements something like a JIT compiler inside and actively allocates executable memory can have such vulnerability. This seems to exclude thumbnail generators written in C, since there is no reason to use JIT in C for thumbnails. Moreover, executable memory should be readonly if handled properly (and in fact, this is a mandatory requirement on many systems, for programs with JIT inside to run at all), and the typical practice is to lock executable memory when you are not editing it. This insulates the possibility of injecting code, even if memory is used incorrectly by mistake. Finally, even if all those protection fails, and the file-scanning process does modify executable memory in an unintended way, how can it possibly be made to do it in the way the malware wants? After all, it is a bug, so it is usually something unpredictable. I just could not imagine how a bug can cause a program to load code byte-by-byte from a file in its entirety and accurately, unless one is intentionally doing this.
Could anyone explain (possibly with examples) how a bug can cause a program to have such probabilistically unlikely behaviour?
