IOEngine

IOEngine is a minimal Event-Condition-Action logic engine driving state transitions by triggering state-changing actions defined in an IOMap. IOEngine drives an IOMap, similar to a multi-step process engine. In this incarnation, the engine is not only minimal but also minimalistic - it will intentionally remain as minimal as possible, that is able to run on a Raspberry Pi Pico with 256K of memory.

Currently, the IOEngine is set up as a 'visible logic engine' demo. Running 'python fan_engine.py' will generate
several hundred lines of print output. Setting EDEBUG to False will generate 1/10th of the output, but also be about 1/20th as interesting to the logic engine enthusiast.

The basic flow is from a values dictionary binding values to boolean evaluations of conditions using
the values and then to actions using the conditions as triggers to perform the action. The forward mapping 'event' is a change of a value in the values dicionary.

Forward mapping:

Keys in values dict -- used in --> Conditions -- used in --> Action Triggers

Forward mapping also allows for backward mapping, such as when testing/binding changed conditions to their their changed values. As this point, the main purpose of backward mapping is to build the forward mapping structures.

Backward mapping:

Action Triggers -- use --> Conditions -- use --> Keys in values dict

Below is the basic interaction between the IOEngine and IOMapper.

IOEngine Overview

There are two types usage scenarios with an associated engine type: Monitor or Transactor type applications and engines. For either type of engine, a given ruleset can be run in either mode depending on how it will be used. The internal mechanism and flow of control will be very different but the ruleset itself is the same.

At this point, only the Transaction Engine is implemented. For small rulesets ( 20-30 rules ), the results should be identical - not nearly identical, identical. If performance scale-up is needed, the bridge is there.

Monitor Type Applications

Monitor type loads and then operates continuously for many thousand of cycles. It detects changed values and only tests action triggers that may be effected by the change.

Monitors are used for situations where only a few of many values change from one binding cycle to the next. The two fan examples are a monitor type application, implemented with a transaction type engine. For a ruleset of say 20-30 rules, the loop latency should be very close. Loop effects seem to predominate over the bitwise operations in general. [ Need to test ]

Transaction Engine Applications

A transaction engine loads and runs for a few cycles and then terminates. The transaction engine doesn't detect changed values that might effect the action triggers - it retests all action triggers from scratch. It is inherently less efficient than a monitor, but runs smaller in memory [ almost certainly ] and is usually faster for small rulesets ( 10-20 rules over 6 cycles ? ) [ need to test ].

An instance of a Transaction engine may be retained over many cycles and reused with multiple IOMapper instances. In fact, the same instances of IOEngine and IOMapper can be reused with a series different values dict. Most values in the values dictionary will change between one invocation and the next - a simple iomapper.read_into_values should synchronize values in the dictionary.

An example might be an Order Entry system. Most Customer Order data in the values dict will change between load_new_order / initialize sequences, so the upfront synchronization cost is the same as a monitor type.

The ECA-driven Order system first loads the next order and initializes the values dict. Then the engine trundles along for several cycles until customer_credit == 'OK' or order.product.required_units < product.unallocated_inventory , etc. In any case, the ruleset reaches some sort of conclusion ( or not ! ) in a few cycles and then ends.

An issue that arises in a 'bind-first' engine rather than 'bind-on-demand' is big, expensive queries. Across several IOEngine bind cycles, the need to run a large query can detected with conditions that fire an action to execute the query, sort of like a side process of minor binding cycles between major cycles.

For example,

 IF   order.priority==='high' AND
      east_coast.inventory.product.available_units < order.product.required_units
 THEN
      run.west_coast_product_inventory_query   # big and expensive, wait for it.

Another scenario is a chain of Transactor engines, one for each step of a given Customer Order. There might be a Customer Order process:

Customer Service Ruleset --> Financial Ruleset --> Inventory Ruleset --> Shipping Ruleset --> etc.

Different engines, one iomapper for each engine, over many different values dicts. This is something like above, but the end state of the previous cycle can reliably be forwarded to the next cycle and taken from there. One consequence is that the all-important order.status or any other 'derived' value doesn't necessarily have to be trusted - it can be determined by running the process engine for a sequence of rules sets.

The major benefit is consistency and flexibility in the order process logic, which can be loaded from a 'compiled' ruleset file, or potentially a customized ruleset for each customer or type of customer.

Another topic that needs to be developed is 'exit conditions'. There are several types:

some goal value is known or a condition is true
loop limited exceeded
nothing changed between one binding cycle and the next.
an exception, possibly intentionally raised

Message Engines and Other Types

There seem to be classes of applications that don't fit the Monitor/Transactor model, messaging for instance.

The other engine type might be more like an analytic 'policy engine' coordinating multiple ECA engines. Possibly IO queues would be driving the whole thing and these separate engines would be able to interact with the queues in a flexible way. Choreography ? Needs work.

There may be engine customizations based on use-case senarios:

ECA Type - monitor type or transactional type,
Messaging Type - With various types of IO queues driven by different ruleset, like advanced routing.
Constraint Engine Type - something like conflict detection and exception handling/logging capability.

These engines might be configurable build parameters.

The IO Engine factory may be invoked by a 'Process Engine'. The Process Engine might be mediating/juggling between different ioengines.

The IO Engine factory might have multipe iomappers in a NamedDict namespace, ex:

IF 's1.temp' > 's1.temp_max' AND
    's2.temp' > 's2.temp_max'
THEN 'this_is_bad_do_something'

s1 = {'temp':123,
      'temp_max': 100 } in namespace 's1'

s2 = {'temp':94,
      'temp_max': 95 } in namespace 's2'

Probably needs a local namespace, for ex. 'loc.number_of_fans_overheating'

Ways to build:

python defined iomapper, conditions, etc defs passed as params to class at _build time.

python defined IOEngineDef passed as params to class ( *IOEngineDef ).

Take output of ioeng.to_dict and create a python dict def, such as rs1_dict = { result from print(rs1.to_dict }. Switch between multiple ruleset dictionaries.

A JSON file definition of IOEngine rule set, with new instances of IOMapper and Evaluator, something like IOEngine( from_dict ) ==> ioeng.from_dict( json_file_loader.load ()) .

JSON file definition of IOEngine rule, using existing instances of IOMapper and Evaluator that is IOEngine( iomapper, evaluator, from_dict = json_file_loader.load() ).

Note that the iom.values_dict will have state, whatever values it last had at the end of the last ruleset load/run. This might be a ruleset sequence, like:

    Order_validate        ->
    Order_checkstock      ->
    Order_checkcredit     ->
    Order_apply_discounts ->
    etc.

Engine Definitions ( External Structures )

The IOEngine constructor is very flexible, but has a large number of parameters and can be a complicated beast to work with. There needs to be some sort of factory class or function to help determine the usage scenario intended, perhaps with different sets of pre-configured defaults.

mapper: IOMapper	mapping external functions calls, either afferent or efferent.
mapper.values: VolatileDict	source of all values for IOEngine and mapper.
conditions: dict[str,list['Conditions']]	action keys with either List[Condition] ( an AND list of Conditions ) or List[List[Condition]] )( an OR list of AND lists of Conditions.
read_keys: list[str]	list of action keys in IOMapper for read_into_values, can read a subset or superset of source values into the values dict rather than using the default read_keys defined in the IOMapper. Haven't needed at this point. It might be a dictionary of different 'routines' for reading IOMapper.
evaluator: Evaluator	evaluates a condition and returns True/False. Can be subclassed for custom validations.
cond_macros:dict[str,list]	macros for sets of conditions for replacement insertion into action triggers at build time.
conflicts_sets: list[set[str]]	a list of sets of conflicting actions, such as { 'turn_device_off', 'turn_device_on' }
from_dict: dict	a dump of IOEngine external and internal structures store-able to and restore-able from a json file. See note below.
debugging:bool	not implemented yet. This will be the main tool for ruleset debugging.

There a dozen or so internal structures, documented in the IOEngine code.

Ther most important internal structures are the condition set and evaluated conditions ('cond_evals'), which is an integer representing the result of testing the conditions in the condition set against current values in the values dictionary. The condition evaluation integer gets carried forward across binding cycles, representing something like a 'state' of the IOEngine.

For example, say a loan application has the condition set:

Slot	Condition
0	Condition('age' , 'lt', 30)
1	Condition('credit_rating', 'eq' 'Bad')
2	Condition('credit_rating', 'eq' 'OK')	lhs and rhs from iom.values
3	Condition('income', 'gte', 20000)	rhs is python type
4	Condition('income', 'lt', 20000)

Note that the 'name' of Condition('age' , 'lt', 30) is 'Condition 0' - it has no other id. The same applies to rules, The 'name' of the rules in the action_trigger_xref are Action 0, Action 1 , etc.

An action trigger of ( 'approve_loan', 0b01101 ), that is condition set slots (0, 2, 4), would indicate conditions:

Condition('age' , 'lt', 30) AND
Condition('credit_rating', 'eq' 'OK') AND
Condition('income', 'gte', 20000)

An AND match between 0b01101 and the evaluated conditions 0b01101 would trigger the 'approve_loan' action in IOMapper. The parameter 'applicant' in the values dict would provide a master reference to the attributes of the new customer loan application.

Note: On a RP Pico, the IOEngine init was taking close to two Mississippi seconds to build internal structures for a monitor type engine. Loading a saved ruleset from a json file and using the from_dict mechanism to build the engine is faster, while also allowing for persistence and import-ability without dynamic importing.

Handling Conflicts

In principle, action conflicts should be resolved during a conflict resolution phase, such as using simple preferences like device_action_preference_order = [ 'turn_device_on', 'turn_device_off' ] to resolve the conflict, or any number of other strategies.

In practice, all action conflicts should be eliminated with filtering conditions. If an implementation needed to control several independent devices no two of which can be 'on' at the same time, then that is a genuine conflict. But still, make life easier, not more difficult, filter them out. Conflict resolution should be more like exception handling - at the conflict level, the 'triggering event' is likely to be a process exception anyway, i.e. 'Customer Order Not Found' or 'Device Not Responding'.

On the other hand, as the engine is set up, 'condition conflicts' are nearly impossible without egregious specification error. Since all values come from the values dictionary and the dictionary is internally consistent, that is no iomapper spec error, all will be well, at least at the 'visible engine' stage of development.

Deployment Option B

The fan_example.py version of the Cheap Fan demo shows how the basic logic of the action trigger definitions can be directly mapped into conventional Python if <condition> <and/or>: type structures using IOMapper directly, saving about 10K and running much faster on a Pico.

Maybe develop/debug the logic of a ruleset with the IOEngine and deploy a 'controller' implementation in Python/MicroPython ?

See https://en.wikipedia.org/wiki/Rule-based_system