ARCcore.filter library's filter factory function ARCcore.filter.create
constructs a filter object
given optional developer-defined constraints (filter specifications) and
optional custom data transformation (bodyFunction
). The details of constructing a filter object are
discussed in detail in the filter API section.
Internally, a filter object's request
method is implemented as a small and simple data flow pipeline
depicted in the follow ASCII diagram:
Filter.request
+----------------------------------------------+
| responseFilterSpec + |
| | |
request -> [iFSP] -> [bodyFunction] -> [rFSP] -> [oFSP] -> response
| | | | |
| | | | |
+--|---------|---------------------------|-----+
run-time | | |
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
factory | | |
| +< bodyFunctiont |
+< in filter spec out filter spec >+
In order to understand the errors returned by a filter object's request
method, it's good to have a basic understanding
of each pipeline stage's role and responsibilities. This knowledge will allow you quickly ascertain the root cause of most
common mistakes made during development. And, save you many trips into the debugger.
The Filter library's core algorithm is implemented by a generic function called the Filter Specification Process (FSP).
FSP uses metadata provided by a filter specification
to customize its behavior. Depending on the developer-defined input to the filter factory, at least one and up
to three calls the FSP are made by a filter object's request
method when dispatching.
FSP is a generic function that accepts a request object containing two related but distinct data values:
response.result
value.As explained in the previous sections, the implementation of Filter.request
incorporates up to three separate
calls to the FSP function.
Each call to the FSP function executes its generic algorithm and produces a response that is either:
A valid request to the FSP is a request that meets the constraints expressed in an FSP's filter specification object.
A well-formed result from an FSP is a valid request that's been pruned of unknown object branches, and that has had optional default values set for missing inputs by the FSP algorithm.
The FSP algorithm controls precisely what lands where but like an industrial robot has no intrinsic knowledge at all about what its doing or why.
Rather, FSP blindly follows a single complex script that interprets a filter specification as a hierarchical set of directives indicating what to accept as "valid". And, and what is considered to meet the "well-formed" bar on a data namespace by namespace basis.
For example, the following filter spec...
var filterSpec = { ____accept: 'jsNumber' };
... reads "I'll accept as valid a request whose type is numerical. And it's required that the caller specify the value."
And the following filter spec...
var filterSpec = { ____accept: 'jsNumber', ____defaultValue: 0 };
... reads "I'll accept as valid a request whose type is numerical. Or, undefined. If undefined the value zero (0) should be set."
One way of thinking about it is to consider the FSP request
generically as a simple JSON document
(plus JavaScript function references as need - we use these for in-process subsystem communication not just JSON)
and the filter specification object as a simple rule set used to (a) verify and then (b) normalize the request data.
Each of the up to three FSP invocations that occur during dispatch of a call to a filter object's request
method
performs has a specific role and set of responsibilities described in the follow subsections.
The input Filter Spec Processor (iFSP) is responsible for ensuring that input passed to request
by a caller
is valid and is normalized (i.e. any default values specified by the inputFilterSpec
provided to the filter
factory) before it passed on to the next stage of the request pipeline.
If the input is invalid, the iFSP returns an error to the caller without passing any data to the next pipeline
stage. This means that if the filter object was constructed with a developer-defined bodyFunction
that this
function will never be called unless the iFSP has completed its validation/normalization without error.
The Response Filter Specification Processor (rFSP) is not under developer control and is responsible for
verifying that the the response
object that is passed through the pipeline stages during request
dispatch
has the correct signature.
Specifically, rFSP ensures that a developer-defined bodyFunction
follows the library's convention for
reporting errors and result(s).
To review, developer-defined bodyFunction
's must adhere to the following conventions:
error
and result
with the following semantics:
error
must be a string value. If complex error information is be communicated, use JSON.error
indicates the result
is invalid and should not be used by the caller.rFSP guarantees that response
has the right "shape" without verifying the value of response.result
.
No normalization of the response
is performed - this is a validation stage only.
The Output Filter Specification Processor (oFSP) is responsible for validating and normalizing the response.result
value of request
calls that claim to have succeeded.
This is not always the case as there may be bugs in the developer-defined bodyFunction
. Or, there may
you might be in the middle of refactoring the code and have changed the outputFilterSpec
definition
without making corollary changes to bodyFunction
. Regardless, if oFSP cannot complete its validation
and normalization of request.result
, then it will clear the value
, set the error
, and return the
error to the caller instead of the proposed result value.
The following subsections discuss factors that were considered during the design of the Filter library.
The design of the filter library was initially motivated by a huge matrix of functions that would need to be implemented in order to realize the design of a large subsystem of the as-yet-to-be-revealed Addressable Resource Class project which briefly deals with data modeling, and software message routing using data semantics.
After starting in at the corner of the matrix and writing several dozen functions and their corollary unit tests it became clear that it would literally take a year to complete one pass. And, given the complexity and the likelihood that the design would need to modified, the likelihood of spinning endlessly without ever really being able to demonstrate anything seemed very high.
Clearly this wasn't going to scale and we needed a way to to eliminate keystrokes, and the necessity of writing tests that were likely to be invalidated and thrown away. And, there was the issue of documentation...
The initial prototype used a little compiler/code generator that read something similar to filter specification and synthesized inline I/O validation code that was consistent and didn't need to be tested (it either all worked or was all broken with fixes easily applied to the code generator).
This worked okay but the solution required a bit of effort to keep track of all the generated artifacts. And, as this approach pushed towards the edges of the function matrix I quickly started running out memory my $5 Digital Ocean/CoreOS test server. Damn it...
So... because of this the code generator was thrown out and replaced by the generic FSP algorithm and
current request
pipeline architecture explained above.
This solved the memory crunch issue by using generic functions to handle the matrix of possibles instead of fully-specialized inline code synthesized by the code generator. This approach was much more difficult to get working as it requires a lot of unit tests (over 500). And, it comes with some runtime overhead. But empirically the overhead does not seem to be much of a problem.
Now, all filter objects regardless of how they are constructed by the filter factory execute the same
generic algorithm that leverages immutable conventions (e.g. function signatures, response object format)
and metadata defined by registered filter specifications to affect
runtime validation, normalization, and error handling for every call to a filter object's request
method.
This means that your application bundle size and runtime memory requirements for using the Filter library are constant; there's no per-filter cost beyond the first use except for the memory consumed by your filter specifications which are typically not very large.
ARCcore/filter is bundled in the ARCcore package which also aggregates the library's internal runtime dependencies: ARCcore.util, ARCcore.types, and ARCcore.identifier. So although ARCcore.filter itself isn't huge, there's little point in unbundling it.
One concern about using the ARCcore package is the package's size; ARCcore v0.0.17 weighs in at ~90K minimized. That's big by JavaScript library standards. In practice the expense appears justified however.
Consistent use of ARCcore.filter in large applications reduces the size of derived applications by eliminating a significant amount of JavaScript code that would normally be inlined in the product to affect data validation, normalization, and error handling.
The design of the Filter library is entirely synchronous; calls to a filter object's request
method
are always handled on the calling thread and completed without yield.
For many use cases this is exactly what's required. However, there are many practical situations where it would be useful to integrate filter functionality into asynchronous processing code.
The original design attempted to handle both cases but was abandoned in favor of the simpler synchronous approach after realizing that the asynchronous constructs can be easily modeled using three synchronous filters:
The Holism application server leverages this pattern extensively. Specifically, Holism Services are filters called by the application server in response to HTTP requests that typically perform asynchronous operations (e.g. accessing external storage subsystems via a network request).
When dispatched, a Holism Service filter's developer-defined bodyFunction
initiates asynchronous operation(s)
returning meaningful information to its caller (Holism app server) synchronously only if the attempt
to initiate the asynchronous operation(s) fails (e.g. database not available). Otherwise, it completes the
request asynchronously by calling either the error or result completion filter constructed by the caller (Holism)
and passed to the service filter via the service filter's input.