The Curiously Recurring In-Place Initialisation Pattern
This page describes various incarnations of a curiously recurring pattern which emerges during our move towards place-based (coordinatised, externalised, address-oriented) programming, away from the traditional orientation towards functions (pure or otherwise) operating on values.
A key, and faulty, idiom underlying both functional and object-oriented programming is that a freshly produced value may not be inspected until it has been fully processed. In OO, this moment is held to be the point at which its constructor (in practice, a mostly ordinary function) has returned. In FP, naturally any value produced within a function may not be inspected until its point of return - any violation of this principle is deemed a "side-effect".
Our new orientation is away from placeless values - every piece of state in a design should be given a more-or-less stable public address, rather than being hidden within impenetrable units of design such as functions or objects. Unfortunately we are faced with the challenge of scaffolding the new world using the techniques of the old, and even when fully incarnated we can't expect a address-oriented architecture to be fully homogeneous - there will always be "impedance boundaries" where it has to interact with traditional architectures, for reasons of efficiency or interoperability.
The places where these "new values" are brought into existence appear rather strange, from the point of view of traditional programming, and we still lack any comprehensive, general techniques for expressing them, either as traditional utilities or else as idioms within Infusion itself.
Prompting Situation
The situation which prompted this article was some work on the BioGaliano map-based species record visualisations, an early prototype of which can be seen at http://biogaliano.org/map-prototype/ . The situation was awkward, and quickly brought into mind numerous other situations where similar issues had cropped up, either within the design of Infusion itself, or even reaching back into the Spring-oriented RSF days of the 00s.
Notably the situation involves interaction with some external, rather unenlightened object-reference oriented library, the https://leafletjs.com/ library currently popular for presenting web-based maps. We have expressed our initialisation of the Leaflet map structure as a member, as so:
fluid.defaults("hortis.leafletMap", {
gradeNames: "fluid.viewComponent",
...
members: {
map: "@expand:L.map({that}.dom.map.0, {that}.options.mapOptions)"
},
...
});
Now, there is a bunch of further Leaflet setup that needs to have occurred with respect to the map before we can proceed to consume the overall component model and use it to render into Leaflet's DOM structures. Under the new post-FLUID-6148 framework, model initialisation occurs "apparently atomically" across a patch of instantiating component tree. This implies that we need to somehow tie the model-oriented consumers of the map into the Leaflet post-processing surrounding the delivery of this value. It may also be helpful to see this situation in the context of the refactoring that led into it - before the consumers of the map were model-oriented, it was sufficient to schedule the Leaflet actions into the component's in the following traditional way:
// Prior, old-fashioned scheduling of Leaflet actions in component's onCreate
fluid.defaults("hortis.leafletMap", {
...
listeners: {
"onCreate.fitBounds": "hortis.leafletMap.fitBounds({that}.map, {that}.options.fitBounds)",
"onCreate.createTooltip": "hortis.leafletMap.createTooltip"
},
...
});
Now under "new rendering" idioms, this is no longer good enough. onCreate is far too late to influence model-based rendering actions, and so we need to be much more specific about when exactly we expect these actions to occur.
In practice, we would like to express that "these actions should occur after the initialisation of the member map, and before any other part of the implementation attempts to access the value of map".
Here the intention expressed in bold is our "in-place conundrum". In a function-oriented, value-oriented, placeless architecture, this intent would be honoured by putting the actions inside the function body that computed "map", after it had been generated by its internal means (a further function call), and before it had been returned to the caller. Unfortunately, in our desired open authorial idiom, the elimination of these unaddressable, "dead spaces" in our designs is our main goal - see the following functional snippet for a sketch .
// Cursed, functional idiom
function produceMap () {
var map = upstreamProduceMap();
// authorial "dead space"
map.fitBounds(); // etc.
return map;
}
So, what to do? Some initial thoughts centred on the use of fluid.promise.fireTransformEvent, or perhaps more widely, the "gyres" promised in Plan to Abolish Invokers and Events. This seemed pretty annoying, since the function contract expects each member of the transform chain to return the value being generated - we have no desire to create silly wrapping functions for perfectly ordinary (if obnoxiously stateful) functions such as fitBounds or createTooltip.
Further thoughts considered creating some further piece of pseudostate, e.g. "initialisedMap", which depended both on map, and also on the workflow functions we expected to execute before map was ready. This was the kind of dodge we would consider in the old Spring days - if your only tool for scheduling is expressing a data dependency, then why not introduce a piece of fake data whose only purpose is to perform the scheduling?
In practice, none of these solutions are terribly desirable since they don't get at the heart of the issue - that the name (address) they want to use to identify map is exactly the same as the name that the outer consumers do. The designer of fitBounds wants to use exactly the same name to address map as the upstream consumers (the renderer, etc.) since anything else represents a nasty loss of design cohesion - we don't expect every author to have to read the design in detail to determine that initialisedMap represents exactly the same element as "map" only in a different workflow condition.
In the end we went with a stopgap solution, based on observing that no consumers of map expected to consume it before the component's model was initialised. Rather than a "pseudoevent" as operated by fireTransformEvent, we simply attached the workflow functions to a perfectly ordinary event which was, indeed, attached to a piece of pseudostate in the component's model, but at least one which consuming authors would not need to know the name of.
// Adopted, stopgap solution - undesirable with two additional "shadow design" elements,
// an event "buildMap" and some pseudostate "mapInitialised"
fluid.defaults("hortis.leafletMap", {
gradeNames: "fluid.viewComponent",
members: {
map: "@expand:L.map({that}.dom.map.0, {that}.options.mapOptions)"
},
model: {
// abuse of expander for scheduling, actually returns "undefined"
mapInitialised: "@expand:{that}.events.buildMap.fire()",
...
},
events: {
buildMap: null
},
listeners: {
"buildMap.fitBounds": "hortis.leafletMap.fitBounds({that}.map, {that}.options.fitBounds)",
"buildMap.createTooltip": "hortis.leafletMap.createTooltip({that}, {that}.options.markup)"
},
...
}
In the pre-FLUID-6148 framework this would have been unjustifiable abuse of the framework since it was advertised that users should not attempt to fire events before a component's onCreate (although this probably worked in practice under many conditions). After FLUID-6148, the much stronger emphasis on construct-time coding has implied that the use of construct-time events is a blessed technique. However, this is still fundamentally abuse of the intention of the framework and the meaning of its primitives - we have a piece of model state which only ever holds the value undefined whose only purpose is to schedule some side-effects before other values which are known to be in the model are consumed. It's a deeply unsatisfactory compromise.
A few phenomena surround this particular situation - firstly, one might ask, why didn't we simply put the map itself into the model? But in practice, the map itself is quite un-model-like material, it actually represents a "view" element from another framework, and this would have made little improvement on our situation, although we could perhaps instead got the advantage of abusing some other framework primitive such as a modelListener.
A better answer to this question might have stepped back and considered - at what level does Infusion actually solve problems like this at all? And whilst there is some small assistance for dealing with these in-place issues where they occur in fine-grained elements such as free-form component options or model material, these problems are in practice only solved at the level of entire components. This implies that a more appropriate solution would be to upgrade the leaflet "Map" into a "mapHolder" component of its own, whose only purpose was to bear constructional responsibility for it. Which of course is what hortis.leafletMap originally was, before it got larded with additional design responsibility. We could then have shifted the workflow into the onCreate of that component.
This still raises issues as before - we are forced to create further, and more fine-grained design elements than was our original design taste, when these so-called "unforeseen" refactoring challenges arise. A lot of our taste for component granularity is based on efficiency considerations - until we have the FLUID-5304 "Infusion compiler" the run-time costs of extra components are really egregious, but also just at the design level incurring 3 extra levels of nesting in the JSON structure represents an annoying readability constraint.
But in practice this "mapHolder" doesn't even clearly solve our original problem. These workflow functions in practice interact with the map at the "view" level of our own design, and it still becomes no clearer that we should expect the map member of the held component to not be effectively addressable before onCreate of the child component has finished. We could perhaps arrange this to be so by ensuring that the mapHolder was not a modelComponent but again this represents expecting a totally unrealistic insight into the framework's scheduling primitives on the part of the designer.
The Wider Situation
This problem reappears in some guise at every level of Infusion's design - as we mentioned at the outset, from some views it is its key authorial challenge but in practice one that it fails to meet, and offers a splintered array of somewhat mismatched solutions to.
At the level of individual component options, Infusion's original design attempted to deal with this via mergePolicies. These are essentially "reduce" functions, and, being directly inherited from the corresponding notion from FP, don't do anything to address the fundamental issue - as the intermediate arguments pass through the hands of the reducers, they are unnamed in the wider design. They are also not terribly amenable to interception, although with modern namespace+priority options distributions it is at least possible to arbitrarily interpose intermediate values in the reduction chain.
At the finer-grained variety of this kind of issue, FLUID-4930 is an umbrella JIRA for a variety of "retrunking" bugs. "retrunking" is a fairly imprecise name given to patterns of options consumption where a site at a leaf attempts to consume a value of a close sibling or, still more abstractly, a direct parent in the same component. Elementary varieties of this problem look as follows:
fluid.defaults("fluid.tests.retrunking", {
gradeNames: "fluid.component",
arrowGeometry: {
length: "{that}.options.polar.length",
width: 10,
headWidth: 20,
headHeight: 20,
angle: "{that}.options.polar.angle",
start: [100, 100],
end: [100, 200]
},
polar: "@expand:fluid.tests.vectorToPolar({that}.options.arrowGeometry.start, {that}.options.arrowGeometry.end)"
});
Here we have a nested value, arrowGeometry.length whose value depends on polar.length - but in order to consume the latter, we first need to consume arrowGeometry.start - which necessarily restarts the process of evaluating arrowGeometry at a trunk node in the options, hence the name "retrunking". This involves a version of our "in-place" conundrum. If arrowGeometry were a classic, functional value, until we had computed all of its children, it would have to be said to be "in evaluation" - and so in this geometry it is impossible that there could be a classical function computing its value and then returning it.
The framework has been resistant to elementary versions of retrunking bugs like the above for many years (although we should return to how it manages this in detail shortly), but a more elaborate version of the bug surfaced recently after the FLUID-6148 refactoring.
fluid.defaults("fluid.tests.retrunkingII", {
gradeNames: "fluid.component",
dom: "@expand:fluid.identity({that}.options.selectors)",
selectors: {
svg: ".flc-bagatelle-svg",
taxonDisplay: ".fld-bagatelle-taxonDisplay",
autocomplete: ".fld-bagatelle-autocomplete",
segment: ".fld-bagatelle-segment",
phyloPic: ".fld-bagatelle-phyloPic",
mousable: "@expand:fluid.tests.FLUID4930combine({that}.options.selectors.segment, {that}.options.selectors.phyloPic)"
}
});
The setup is extremely similar to the above, but we have left a placeholder for the evaluation of a member "dom" whose dataflow mirrors that of a standard fluid.viewComponent. This sets in motion the process of consuming the entire options tree at "selectors", which amongst itself contains a cyclic retrunking, that is, a member who consumes a sibling via an expander. This ends up confusing the framework's workflow, since the first data demand for the trunk value "selectors" marks the entire tree as "in evaluation", and any further attempts to revisit it encounter this marker rather than the currently populating trunk value.
All this raises rather deep philosophical issues about what it means for a value to be "fully evaluated", especially if that value classically appears to be a container for other values. This clearly has a different meaning depending on the nature of the consumer. If this consumer is a classical function, it naturally expects it to be in the classical sense, and indeed we have a JIRA about this too - FLUID-5981. But if the consumer of the value is the framework's own machinery, the answer is clearly different, otherwise no variety of "retrunking" could be supported. This directly feeds into the "containment without dependency" [footnote - first cited in PPIG 2015 paper, this is the central insight of the 2010 Pupusa Queue Conversation] notion that founded the entire framework. From an OO or FP view of a structure, the structure "depends on" its members, and from our point of view it doesn't.
So - are we "changing" a trunk value (object or hash) by "discovering" further members nested within it during our evaluation process? In our particular riff on the notion of "referential transparency", we are not - since to us, a reference is an address. Having noted that we know what the type of the "value" is, and what its address is, we have noted two unchangeable facts about it - these do not alter during the evaluation process for the "value", nor for any other values that are found to be "nested" within it.
But to return to the theme at the head of the page, implementing such a process of "discovery" in a conventional programming language leads our "functions" to have a very peculiar appearance, and also a rather stereotypical one - they necessarily must accept some kind of more or less massive "environment" as an argument, in order that they may deliver a freshly discovered value "in-place" into its appropriate spatial position at the very earliest possible moment. In terms of the base programming language, this is a huge weakness - the function as well as being massively impure, has a massive environmental dependency that greatly restricts its reuse. It is for this reason that taking the hit of "being an Infusion" is something that we want to incur as rarely as possible, despite our commitment to pluralities in Infusions.
Object Delivery Sites Through The Ages
Let's now survey what these sites actually look like within Infusion, starting at the small scale where individual options values are "delivered":
Delivery of Option Values
Our first tour takes in the site where individual options values are discovered. This part of the code has not changed substantially since the original "ginger world" rewrite of ca. 2012 described in FLUID-4330, but is due for a massive rewrite once we can afford to get going with the FLUID-5304 "Infusion Compiler" - currently at FluidIoC.js line 3099:
fluid.expandSource = function (options, target, i, segs, deliverer, source, policy, recurse) {
var expanded, isTrunk;
var thisPolicy = fluid.derefMergePolicy(policy);
if (typeof (source) === "string" && !thisPolicy.noexpand) {
if (!options.defaultEL || source.charAt(0) === "{") { // hard-code this for performance
fluid.pushActivity("expandContextValue", "expanding context value %source held at path %path", {source: source, path: fluid.path.apply(null, segs.slice(0, i))});
expanded = fluid.copy(fluid.resolveContextValue(source, options));
fluid.popActivity(1);
} else {
expanded = source;
}
}
else if (thisPolicy.noexpand || fluid.isUnexpandable(source)) {
expanded = fluid.copy(source);
}
else if (source.expander) {
expanded = fluid.expandExpander(deliverer, source, options);
}
else {
expanded = fluid.freshContainer(source);
isTrunk = true;
} // 2nd branch was another partial site for FLUID-6428 but no longer seems to be needed
if (expanded !== fluid.NO_VALUE /* && source !== undefined */) {
deliverer(expanded);
}
if (isTrunk) {
recurse(expanded, source, i, segs, policy);
}
return expanded;
};
Note the use of "deliverer" at line 3122. This classically fits the pattern we are describing - the fully expanded value must be delivered in-place immediately since we are just about to recurse on further nested values which may end up referring to it - yet this function can't be expected to know its address. This is by now one of the most hated code sites in the framework and one of the most classically optimised away once we build FLUID-5304.
Delivery of Component Values
Next we turn to the wider scale - exactly the same phenomenon occurs with respect to entire components. One of the key realisations in the (current) FLUID-6148 rewrite is that the site of component delivery needed to become completely focused - that is, the minimum time must elapse between construction of the actual component reference and its registration in place in the component tree. Here is the setup for this in the previous world (with the basic structure of the ca. 2012 framework as above, but tweaked in mid-2015, actually at the time of writing, March 2020, still in current master):
fluid.initDependent = function (that, name, localRecord) {
if (that[name]) { return; }
....
else if (component.type) {
var type = fluid.expandImmediate(component.type, that, localDynamic);
if (!type) {
fluid.fail("Error in subcomponent record: ", component.type, " could not be resolved to a type for component ", name,
" of parent ", that);
}
var invokeSpec = fluid.assembleCreatorArguments(that, type, {componentRecord: component, memberName: name, localDynamic: localDynamic});
instance = fluid.initSubcomponentImpl(that, {type: invokeSpec.funcName}, invokeSpec.args);
}
// NOTE: 2012 framework here classically had - that[name] = instance
// but in 2015 this was found to be far too late and shifted halfway up the stack, see below
return instance;
};
// This goes upstairs to fluid.initSubcomponentImpl, where we bizarrely remain in Fluid.js for quite a while, in deference to the mythical "non-IoC components"
fluid.initSubcomponentImpl = function (that, entry, args) {
var togo;
if (typeof (entry) !== "function") {
var entryType = typeof (entry) === "string" ? entry : entry.type;
togo = entryType === "fluid.emptySubcomponent" ?
null : fluid.invokeGlobalFunction(entryType, args);
} else {
togo = entry.apply(null, args);
}
return togo;
};
// Which hits the creator function, via fluid.makeComponentCreator
fluid.initComponent = function (componentName, initArgs) {
var options = fluid.defaults(componentName);
if (!options.gradeNames) {
fluid.fail("Cannot initialise component " + componentName + " which has no gradeName registered");
}
var args = [componentName].concat(fluid.makeArray(initArgs));
var that;
fluid.pushActivity("initComponent", "constructing component of type %componentName with arguments %initArgs",
{componentName: componentName, initArgs: initArgs});
that = fluid.invokeGlobalFunction(options.initFunction, args);
...
that.events.onCreate.fire(that);
fluid.popActivity();
return that;
};
// Which hits an old-fashioned "concrete creator function" e.g. fluid.initLittleComponent - FINALLY WE GET TO THE POINT WHERE THE REFERENCE IS CONSTRUCTED!
fluid.initLittleComponent = function (name, userOptions, localOptions, receiver) {
var that = fluid.typeTag(name);
that.lifecycleStatus = "constructing";
localOptions = localOptions || {gradeNames: "fluid.component"};
that.destroy = fluid.makeRootDestroy(that); // overwritten by FluidIoC for constructed subcomponents
var mergeOptions = fluid.mergeComponentOptions(that, name, userOptions, localOptions);
mergeOptions.exceptions = {members: {model: true, modelRelay: true}}; // don't evaluate these in "early flooding" - they must be fetched explicitly
var options = that.options;
that.events = {};
// deliver to a non-IoC side early receiver of the component (currently only initView)
(receiver || fluid.identity)(that, options, mergeOptions.strategy);
fluid.computeDynamicComponents(that, mergeOptions);
// TODO: ****THIS**** is the point we must deliver and suspend!! Construct the "component skeleton" first, and then continue
// for as long as we can continue to find components.
for (var i = 0; i < mergeOptions.mergeBlocks.length; ++i) {
mergeOptions.mergeBlocks[i].initter();
}
mergeOptions.initter();
delete options.mergePolicy;
fluid.instantiateFirers(that, options);
fluid.mergeListeners(that, that.events, options.listeners);
return that;
};
// Still in Fluid.js, we are about to bat back to FluidIoC.js for "expandComponentOptions"
fluid.mergeComponentOptions = function (that, componentName, userOptions, localOptions) {
var rawDefaults = fluid.rawDefaults(componentName);
var defaults = fluid.getMergedDefaults(componentName, rawDefaults && rawDefaults.gradeNames ? null : localOptions.gradeNames);
var sharedMergePolicy = {};
var mergeBlocks = [];
if (fluid.expandComponentOptions) {
mergeBlocks = mergeBlocks.concat(fluid.expandComponentOptions(sharedMergePolicy, defaults, userOptions, that));
}
....
}
// Here we go - in fluid.expandComponentOptions we prepare to report the component's existence seemingly as an irrelevant side-effect
// This is the initial entry point from the non-IoC side reporting the first presence of a new component - called from fluid.mergeComponentOptions
fluid.expandComponentOptions = function (mergePolicy, defaults, userOptions, that) {
var initRecord = userOptions; // might have been tunnelled through "userOptions" from "assembleCreatorArguments"
var instantiator = userOptions && userOptions.marker === fluid.EXPAND ? userOptions.instantiator : null;
fluid.pushActivity("expandComponentOptions", "expanding component options %options with record %record for component %that",
{options: instantiator ? userOptions.mergeRecords.user : userOptions, record: initRecord, that: that});
if (!instantiator) { // it is a top-level component which needs to be attached to the global root
instantiator = fluid.globalInstantiator;
initRecord = { // upgrade "userOptions" to the same format produced by fluid.assembleCreatorArguments via the subcomponent route
mergeRecords: {user: {options: fluid.expandCompact(userOptions, true)}},
memberName: fluid.computeGlobalMemberName(that),
instantiator: instantiator,
parentThat: fluid.rootComponent
};
}
that.destroy = fluid.fabricateDestroyMethod(initRecord.parentThat, initRecord.memberName, instantiator, that);
instantiator.recordKnownComponent(initRecord.parentThat, that, initRecord.memberName, true);
var togo = expandComponentOptionsImpl(mergePolicy, defaults, initRecord, that);
fluid.popActivity();
return togo;
};
// And HERE, 3 function calls deeper, we finally attach the component to its parent inside instantiator.recordKnownComponent
that.recordKnownComponent = function (parent, component, name, created) {
parent[name] = component;
if (fluid.isComponent(component) || component.type === "instantiator") {
var parentPath = that.idToShadow[parent.id].path;
var path = that.composePath(parentPath, name);
recordComponent(parent, component, path, name, created);
that.events.onComponentAttach.fire(component, path, that, created);
} else {
fluid.fail("Cannot record non-component with value ", component, " at path \"" + name + "\" of parent ", parent);
}
};
The above trace shows an incredible rigmarole spanning 7 functions. Classically, the value of the component is demanded in fluid.initDependent at the base of the stack. The original 2012-era framework waited for the component to be completely constructed and control to return all the way back through the 7 functions until the reference was attached to its parent. This is what a well-behaved object-oriented framework would do. The 2015 rewrite was structurally conservative, and retained most of the function scaffolding intact, but it was morally progressive, since the value was now "delivered" in-place to the instantiator's records half-way up the stack, whilst it was partway through evaluation. However, the resulting structure is a shambolic and confused mass of old and new intentions - however, note that, crucially, we do succeed in attaching the component before "anything important" has happened to it, in particular the call to expandComponentOptionsImpl which will attempt to resolve references attached to it \[ footnote - Note that the crucial midpoint of this refactoring occurred during the following commit of Feb 11th 2015 - https://github.com/fluid-project/infusion/commit/a5a74f3f164db63641ef8c06c16a8f13a86efdb4#diff-50cd32a7983e921376821a0476122713 as part of the general rationalisation eliminating demands blocks (finally achieved in April) and the storage of records a single "global instantiator"]
Contrast this with the incredibly focused post-FLUID-6148 framework in which a component "shell" is constructed and attached immediately via instantiator.recordKnownComponent, before any further evaluation of any kind has begun:
/**
* Creates the shell of a component, evaluating enough of its structure to determine its grade content but
* without creating events or (hopefully) any side-effects
*
* @param {Potentia} potentia - Creation potentia for the component
* @param {LightMerge} lightMerge - A set of lightly merged component options as returned from `fluid.lightMergeRecords`
* @return {Component|Null} A component shell which has begun the process of construction, or `null` if the component
* has been configured away by resolving to the type "fluid.emptySubcomponent"
*/
fluid.initComponentShell = function (potentia, lightMerge) {
....
var that = lightMerge.type === "fluid.emptySubcomponent" ? null : fluid.typeTag(lightMerge.type, potentia.componentId);
if (that) {
that.lifecycleStatus = "constructing";
instantiator.recordKnownComponent(parentThat, that, memberName, true);
// mergeComponentOptions computes distributeOptions which is essential for evaluating the meaning of shells everywhere
var mergeOptions = fluid.mergeComponentOptions(that, potentia, lightMerge);
that.events = {};
}
return that;
};
In functional terms, it is still nuts, but at least much smaller nuts, and shows the very clearest expression of our notion that "place" logically precedes almost any other aspect of a component's identity - if the language permitted it, we would talk about the component's place before we even produced the object reference which corresponds to it. Our components "grow in place" like seeds planted in soil, rather than placeless values excreted into a void by atmospheric gasbags. Having "the same component" somewhere else other than where it was created has no meaning - if we want it somewhere else, we must tear down the original and reseed it in the new site, and have it grow in the new conditions - where it may well grow into something quite different.