ARCcore.filter library's filter factory function ARCcore.filter.create constructs a filter object given optional developer-defined constraints (filter specifications) and optional custom data transformation (bodyFunction). The details of constructing a filter object are discussed in detail in the filter API section.

Internally, a filter object's request method is implemented as a small and simple data flow pipeline depicted in the follow ASCII diagram:

         Filter.request
         +----------------------------------------------+
         |           responseFilterSpec +               |
         |                              |               |
request -> [iFSP] -> [bodyFunction] -> [rFSP] -> [oFSP] -> response
         |  |         |                           |     |
         |  |         |                           |     |
         +--|---------|---------------------------|-----+
run-time    |         |                           |
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
factory     |         |                           |
            |         +< bodyFunctiont            |
            +< in filter spec    out filter spec >+

In order to understand the errors returned by a filter object's request method, it's good to have a basic understanding of each pipeline stage's role and responsibilities. This knowledge will allow you quickly ascertain the root cause of most common mistakes made during development. And, save you many trips into the debugger.

Filter Specification Processors (FSP)

The Filter library's core algorithm is implemented by a generic function called the Filter Specification Process (FSP). FSP uses metadata provided by a filter specification to customize its behavior. Depending on the developer-defined input to the filter factory, at least one and up to three calls the FSP are made by a filter object's request method when dispatching.

FSP Implementation Details

FSP is a generic function that accepts a request object containing two related but distinct data values:

• a filter spec object (deserialized JSON) that specializes the behavior of the FSP algorithm
• some other value that we'll pass through the specialized FSP algorithm to produce a response.result value.

As explained in the previous sections, the implementation of Filter.request incorporates up to three separate calls to the FSP function.

Each call to the FSP function executes its generic algorithm and produces a response that is either:

• an error indicating an invalid request
• a well-formed result

Valid Requests & Well-Formed Responses

A valid request to the FSP is a request that meets the constraints expressed in an FSP's filter specification object.

A well-formed result from an FSP is a valid request that's been pruned of unknown object branches, and that has had optional default values set for missing inputs by the FSP algorithm.

The FSP algorithm controls precisely what lands where but like an industrial robot has no intrinsic knowledge at all about what its doing or why.

Rather, FSP blindly follows a single complex script that interprets a filter specification as a hierarchical set of directives indicating what to accept as "valid". And, and what is considered to meet the "well-formed" bar on a data namespace by namespace basis.

For example, the following filter spec...

var filterSpec = { ____accept: 'jsNumber' };

... reads "I'll accept as valid a request whose type is numerical. And it's required that the caller specify the value."

And the following filter spec...

var filterSpec = { ____accept: 'jsNumber', ____defaultValue: 0 };

... reads "I'll accept as valid a request whose type is numerical. Or, undefined. If undefined the value zero (0) should be set."

One way of thinking about it is to consider the FSP request generically as a simple JSON document (plus JavaScript function references as need - we use these for in-process subsystem communication not just JSON) and the filter specification object as a simple rule set used to (a) verify and then (b) normalize the request data.

FSP Roles & Responsibilities

Each of the up to three FSP invocations that occur during dispatch of a call to a filter object's request method performs has a specific role and set of responsibilities described in the follow subsections.

Input Filter Specification Processor (iFSP)

The input Filter Spec Processor (iFSP) is responsible for ensuring that input passed to request by a caller is valid and is normalized (i.e. any default values specified by the inputFilterSpec provided to the filter factory) before it passed on to the next stage of the request pipeline.

If the input is invalid, the iFSP returns an error to the caller without passing any data to the next pipeline stage. This means that if the filter object was constructed with a developer-defined bodyFunction that this function will never be called unless the iFSP has completed its validation/normalization without error.

Response Filter Specification Processor (rFSP)

The Response Filter Specification Processor (rFSP) is not under developer control and is responsible for verifying that the the response object that is passed through the pipeline stages during request dispatch has the correct signature.

Specifically, rFSP ensures that a developer-defined bodyFunction follows the library's convention for reporting errors and result(s).

To review, developer-defined bodyFunction's must adhere to the following conventions:

• Accepts a single value as input.
• Returns an object with sub-properties error and result with the following semantics:
  • If set, error must be a string value. If complex error information is be communicated, use JSON.
  • If set, error indicates the result is invalid and should not be used by the caller.

rFSP guarantees that response has the right "shape" without verifying the value of response.result. No normalization of the response is performed - this is a validation stage only.

Output Filter Specification Processor (oFSP)

The Output Filter Specification Processor (oFSP) is responsible for validating and normalizing the response.result value of request calls that claim to have succeeded.

This is not always the case as there may be bugs in the developer-defined bodyFunction. Or, there may you might be in the middle of refactoring the code and have changed the outputFilterSpec definition without making corollary changes to bodyFunction. Regardless, if oFSP cannot complete its validation and normalization of request.result, then it will clear the value, set the error, and return the error to the caller instead of the proposed result value.

Architecture Considerations

The following subsections discuss factors that were considered during the design of the Filter library.

Application Bundle Size

The design of the filter library was initially motivated by a huge matrix of functions that would need to be implemented in order to realize the design of a large subsystem of the as-yet-to-be-revealed Addressable Resource Class project which briefly deals with data modeling, and software message routing using data semantics.

After starting in at the corner of the matrix and writing several dozen functions and their corollary unit tests it became clear that it would literally take a year to complete one pass. And, given the complexity and the likelihood that the design would need to modified, the likelihood of spinning endlessly without ever really being able to demonstrate anything seemed very high.

Clearly this wasn't going to scale and we needed a way to to eliminate keystrokes, and the necessity of writing tests that were likely to be invalidated and thrown away. And, there was the issue of documentation...

The initial prototype used a little compiler/code generator that read something similar to filter specification and synthesized inline I/O validation code that was consistent and didn't need to be tested (it either all worked or was all broken with fixes easily applied to the code generator).

This worked okay but the solution required a bit of effort to keep track of all the generated artifacts. And, as this approach pushed towards the edges of the function matrix I quickly started running out memory my $5 Digital Ocean/CoreOS test server. Damn it...

So... because of this the code generator was thrown out and replaced by the generic FSP algorithm and current request pipeline architecture explained above.

This solved the memory crunch issue by using generic functions to handle the matrix of possibles instead of fully-specialized inline code synthesized by the code generator. This approach was much more difficult to get working as it requires a lot of unit tests (over 500). And, it comes with some runtime overhead. But empirically the overhead does not seem to be much of a problem.

Now, all filter objects regardless of how they are constructed by the filter factory execute the same generic algorithm that leverages immutable conventions (e.g. function signatures, response object format) and metadata defined by registered filter specifications to affect runtime validation, normalization, and error handling for every call to a filter object's request method.

This means that your application bundle size and runtime memory requirements for using the Filter library are constant; there's no per-filter cost beyond the first use except for the memory consumed by your filter specifications which are typically not very large.

ARCcore/filter is bundled in the ARCcore package which also aggregates the library's internal runtime dependencies: ARCcore.util, ARCcore.types, and ARCcore.identifier. So although ARCcore.filter itself isn't huge, there's little point in unbundling it.

One concern about using the ARCcore package is the package's size; ARCcore v0.0.17 weighs in at ~90K minimized. That's big by JavaScript library standards. In practice the expense appears justified however.

Consistent use of ARCcore.filter in large applications reduces the size of derived applications by eliminating a significant amount of JavaScript code that would normally be inlined in the product to affect data validation, normalization, and error handling.

Synchronous vs Asynchronous

The design of the Filter library is entirely synchronous; calls to a filter object's request method are always handled on the calling thread and completed without yield.

For many use cases this is exactly what's required. However, there are many practical situations where it would be useful to integrate filter functionality into asynchronous processing code.

The original design attempted to handle both cases but was abandoned in favor of the simpler synchronous approach after realizing that the asynchronous constructs can be easily modeled using three synchronous filters:

• One to initiate the asynchronous operation.
• One to handle error(s) that may occur if the asynchronous operation fails.
• One to handle result(s) when the asynchronous operation succeeds.

The Holism application server leverages this pattern extensively. Specifically, Holism Services are filters called by the application server in response to HTTP requests that typically perform asynchronous operations (e.g. accessing external storage subsystems via a network request).

When dispatched, a Holism Service filter's developer-defined bodyFunction initiates asynchronous operation(s) returning meaningful information to its caller (Holism app server) synchronously only if the attempt to initiate the asynchronous operation(s) fails (e.g. database not available). Otherwise, it completes the request asynchronously by calling either the error or result completion filter constructed by the caller (Holism) and passed to the service filter via the service filter's input.