I know that modern processors support memory caching and am fairly familiar with it, but the problem with it (from my point of view) is that caching is opaque to the programmer. It’s something that the processor decides to do on the fly based on locality and recently accessed memory. Which is fine, but there are many, many cases where it would be reasonable for the programmer to be able to say “I want block X to be nearer the processor than block Y” – but there is no way of explicitly saying this. Ultimately, what I’d like to see is something like the following: an array of processors laid out in a grid surrounded by local memory (excluding on-processor cache), and a block of global memory out somewhere. One processor could access any memory anywhere, but the further out it goes, the slower access is. Feasible? I don’t know. But I’m trying to imagine what computers would look like if they were actually designed to multitask, rather than hacks around a uniprocessing machine to make it multiprocess.

Memory consistency is a difficult problem but not insoluble. But I think a general rule of thumb (for what I’m thinking of) is that if you need to access the same piece of memory frequently, you should not be a different process. If you’re modifying system-wide data, do so through the operating system processor via messaging (or something like the Pi calculus). In any case, at the very least you could serialize data going to the operating system for system-wide operations, which while introducing potential race conditions would at least keep memory consistent, and this operating system could message individual processors to update their cache in the event of a change. You get around shared data by having specific processors designated to hand shared tasks (an operating system and helpers, if you will).

Again, this is me bloviating on how I might multiprocess if I were to re-build the computer with multiprocessing in mind.

As an example, suppose that processor A runs task foo that accesses and modifies data at system-wide memory location 0×80001436. Because all processors in the system have a 64-byte cache line, it loads all system-wide memory from addresses 0×80001400 through 0×80143e. Before it gets a chance to actually make the modification, Processor A stops executing task foo.

Shortly thereafter, Processor B schedules task foo, loads the memory, performs the memory modification, and writes the cache line back to system-wide memory before moving on to another task.

Later, Processor A starts running task foo again, which now needs to use the data stored at system-wide location 0×80143e. Processor A sees that it still has that memory cached, so it uses its local cache. Unfortunately, that was cached prior to Processor B executed parts of task foo, so Processor A doesn’t see the modified contents of system-wide location 0×80143e. So Processor A executes task foo based on out-of-date data.

That’s just one problem. Other potential problems introduced by memory caching include the following:

1. Processor B modifies data in its cache but does not write it back to system-wide memory before Processor A accesses the same data. Again, Processor A executes a task based on obsolete data.

2. Both Processor A and Processor B load the same cache line. After both loads, both processors modify a different part of the cache line, then write the whole thing back out to system-wide memory. Unless some mechanism exists to prevent it, the modifications written out by the first processor will be lost when the second processor writes its version of the same cache-line to system-wide memory.

Some of those problems are actually handled by the microcode designed into many modern processors. Some of them have to be carefully handled by software. (Fortunately, much of that work usually falls to the operating system rather than individual applications.

I’d also point out that the vast majority of methods used to communicate between processors involve memory. There are other possibilities for inter-processor communications (such as using direct signals), but those tend to come with problems of their owns, especially as they become complex enough to communicate any useful data.

Even shared peripherals — such as both processors displaying data to the same monitor or accessing the same hard drive — come with their own issues.

Well, yes, but that has to remain the case to some degree. After all, if multiple processors didn’t share the same memory, then any time a process got loaded on a different processor, then the new processor’s memory would have to be reloaded with that processor’s application space. And even then, this implies that there’s a common storage space shared by all processors. Otherwise, there’d be no way to move a process to a different processor. At that point, you might as well just be running your processes each on a separate machine.