...
Whilst there may be ways around some of these problems, the discussion so far doesn't shed much light on what we could do about the workflow ("sandwich") itself. Cindy has noted in the channel that it can seem a little odd, to someone who has to deal with it quite closely - in particular, the fact that defaults merging represents "merging before expansion" whilst the merging applied to options distribution represents "merging after expansion". A big part of the reason we did defaults merging the way we did was in order to allow for the possibility of caching large parts of the workflow, and preferably in a form which was JSON-amenable. This was primarily done in the bad old CSpace days of 2009, where we were faced with a very large page comprising perhaps a thousand Infusion components which needed to render relatively quickly on the still rather rickety JS VMs of the day. This also drove the "accumulative linearisation" approach which we abandoned the form of in the FLUID-5887 fix (although we noted that we are still actually following exactly the same algorithm, if a little more inefficiently, as a result of its "painter's-like" nature). In the FLUID-4982 world, we plan for "huge chains of linked immutable prototypes" as our optimisation for grade merging and loading - which hardly seem amenable to effective JSON representation. If we can make this work, it seem that completely abolishing phase 1 of the merging and expansion process will be feasible in a performant system. This will also return it more towards accepted computer science norms, where the necessity for "expansion before merging" (that is, evaluation of function arguments before returning results) is taken as axiomatic. This might also simplify the requirements on the "mergePolicy hoisting system" mentioned in the previous section.
We need massive improvements in performance during component instantiation, which has not been heavily optimised since the CSpace days. Our optimisation work on invokers and listeners for FLUID-5249 and FLUID-5796 showed the effectiveness of the "monomorphisation" approach recommended by Egorov and others for improving the performance of code governed by the "polymorphic inline caches" invented by Ungar et al. for accelerating highly dynamic languages. We need to do something similar for the much more complex expansion and merging process. It seems we should be able to reuse some of the same ideas in a more elaborate form - for example, the "shape cache" for the V8 runtime is a simple flat model of property occurrence on a single object. The merging process features "occlusion" of earlier expanded objects by later ones, but if we can be sure that successive resolved objects are the same "shape" in the deep sense - that is, as entire JSON objects, we can be sure that the resulting merged object fetches material from its source in the same pattern. This leads to the idea of a form of symbolic execution of the merge pathway. Having acquired the shape of fetched items, one imagines that one would replace them by "self-skeletons" - objects which have their own path references replacing all leaves, and executing the merge operation with those instead. Or instead, to perform the merge with "tokens" wrapping both trunk and leaf objects which will allow provenance information to be tracked. This "symbolic merge output" would be "keyed" in storage by not only the list of grades involved, but also by the shapes of every object fetched via expanders or references. Options distributions and user records would be assigned "nonce grade names" in order to participate in this tracking. After that, merging simply has the form of i) fetching this record, ii) scanning through the monomorphised list of unoccluded expansion material, fetching the root references, iii) writing the unoccluded parts of these objects into a single "stencil object" which contains all variable material for the finally merged options in a single layer. All other material would be sourced from the static "liana chains of immutable prototypes" which we use to represent the defaults chain.
This leaves a couple of questions - firstly, what becomes of the "partial evaluation" model? The result is that "a lot more work" gets done during the minimal phase of the algorithm in that at the very moment we determine the final grade lists around the skeleton, we also get all information resulting from the immutable prototype chains for defaults for free. This is fine - we then just need to schedule the work of fetching the unoccluded expansion material in the familiar ginger way, once we have been signalled to start the process.
Secondly, what of mergePolicies? If we require fully "provenanced" output, does this imply that we need to prohibit all mergePolicies which do anything other than shift and select object references (the "path-based model transform dialect without transforms" that we started with in 2011)? Perhaps not. We could just distinguish between two classes of mergePolicies - i) the default, which selects only unoccluded material, and ii) anything else, which might expect to read ALL material. In the latter case, we automatically assemble each participating non-empty element into an array, WHILST tracking its provenance, and then only operate the mergePolicy at the very final step. This also has the virtue of preserving our original "reduce-based" mergePolicies and placing them on a sane footing - in fact they will only ever reduce arrays, and only ever see fully expanded material, and only ever operate directly back-to-back rather than in a scattered fashion throughout the expansion process.
This all seems generally workable, subject to getting some detailed understanding of how the "stencilling" process that we use to symbolically compute occluding contours during the merge process will actually work.