If you remember our previous part 1 article we highlighted some of the inherent complexities we introduce by writing code the way we do today. We also stopped at a high level because (frankly speaking) this was going to be a very long read. We had to stop ranting about the current state of affairs :(( In that article, we came to the conclusion that maybe we needed a way to remove those unnecessary complexities and deal with them outside the data flow graph to achieve maximum flexibility and speed.
If you think about our software industry today all we do , day in and day out, is “transform” data from one state to another. The systems we have in place today can be seen , at a very high level, as black boxes that take a variable number of arguments as inputs and produce a well defined output The purpose of all programs, and all parts of those programs, is to transform data from one form to another.
If you remember our previous part 2 article we used a central memory location to share data between two separate threads each running either in the same process or across separate processes. These processes could even be located on separate machine or even be written in two different languages. From the above you can see that we only have one thread writing to the shared data. The other thread is only reading this shared data once it is notified that the data was correctly published by the writer thread.
As per our previous part 1 article we identified that we need a way to serialize objects into memory in a very efficient manner. To recap here is a picture showing the steps we need to remove when sending/receiving objects over the network. FTR we are still not interested in solving how these troublesome bytes are transferred over the network from one process to another once the data is ready to be transferred.
One of the topics that you often have to deal with in java, especially in high performance and low latency distributed code, is how you send a set of objects to a worker thread in the most efficient manner. This thread could be running in the same process or on a completely separate machine located somewhere else in your network. This has always been the case for people coming from a c/c++ environment even before the advent of Java.