mmx metadata framework
...the DNA of your data
MMX metadata framework is a lightweight implementation of OMG Metadata Object Facility built on relational database technology. MMX framework
is based on three general concepts:
Metamodel | MMX Metamodel provides a storage mechanism for various knowledge models. The data model underlying the metadata framework is more abstract in nature than metadata models in general. The model consists of only a few abstract entities... see more.
Access layer | Object oriented methods can be exploited using inheritance to derive the whole data access layer from a small set of primitives created in SQL. MMX Metadata Framework provides several diverse methods of data access to fulfill different requirements... see more.
Generic transformation | A large part of relationships between different objects in metadata model are too complex to be described through simple static relations. Instead, universal data transformation concept is put to use enabling definition of transformations, mappings and transitions of any complexity... see more.

MMX Framework: Extending with Extended Properties

December 10, 2010 13:36 by mmx

Pretty often there is a need to add some extra attributes to a metamodel that has already been deployed. The need for those changes usually arises from changed requirements for an application. So how to enable those changes without extending the metamodel (M2 layer) itself?

There is a special property type, AnyProperty, to handle this scenario. Whan an object type (class) has a property of type AnyProperty, an object (instance of the class) can have an arbitrary number of different properties of type AnyProperty that all share the same property_type_cd. Then how to distinguish between those properties? 

All the instances of an AnyProperty descend from the same property type of a class in M2 layer, but an application who 'owns' them can give them new names. Alternatively, it is possible to use the same name, but a distinguishing prefix directly in the value field of the property. So an AnyProperty instance 'belongs' to an application and its completely up to the application how it chooses to use, denote and present this property.

Downsides (there obviously are some, right?). First, the inheritance mechanisms built into MMX Framework (MMXMD API) obviously cannot provide full support to those properties, as the names of them are not known to M2 layer. Second, AnyProperty is defined as String in M2 so an application cannot rely on the type information if it chooses to have an instance of AnyProperty that is implemented with some other datatype (eg. an XML document or a date formatted as a string).

Anyway, this mechanism (which bears some resemblance with anyAttribute element in XML Schema) enables making run-time changes in metamodel when design-time is not an option (run-time and design-time here refer to modeling, not application design), just remember to use it with care.

 



XDTL Packages in Five Easy Steps: Beginner

November 14, 2010 13:24 by mmx

Continuing from the previous step, this one actually does something useful, namely transfers some stuff from one location to another.

<?xml version="1.0" encoding="UTF-8"?>
<xdtl:package name="loader.test"
  xmlns:xdtl="http://xdtl.org/xdtl" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://xdtl.org/xdtl http://xdtl.org/xdtl/xdtl.xsd">

  <xdtl:parameter name="filename" required="1"></xdtl:parameter>
  <xdtl:parameter name="tablename" required="1"></xdtl:parameter>
  <xdtl:parameter name="dbhost" default="localhost:5432" required="0"></xdtl:parameter>
  <xdtl:parameter name="dbname" default="postgres" required="0"></xdtl:parameter>
  <xdtl:parameter name="dbuser" default="postgres" required="0"></xdtl:parameter>
  <xdtl:parameter name="dbpass" default="" required="0"></xdtl:parameter>

  <xdtl:variable name="tmpdir">/tmp/incoming</xdtl:variable>
  <xdtl:variable name="bakdir">/tmp/incoming/bak</xdtl:variable>
  <xdtl:variable name="nocr">-e 's/\r//g'</xdtl:variable>
  <xdtl:variable name="tempdb">staging</xdtl:variable>

  <xdtl:connection name="local" type="DB">jdbc:postgresql://$dbhost/$dbname?user=$dbuser&amp;password=$dbpass&amp</xdtl:connection>

  <xdtl:tasks>
    <xdtl:task name="load_file">
      <xdtl:steps>
        <xdtl:get source="ftp://www.site.com/$filename.zip" target="$tmpdir" overwrite="1"/>
        <xdtl:unpack source="$tmpdir/$filename.zip" target="$tmpdir" overwrite="1"/>
        <xdtl:strip expr="$nocr" source="$tmpdir/$filename.txt" target="$tmpdir/$filename.tmp" overwrite="1"/>
        <xdtl:read source="$tmpdir/$filename.tmp" target="$tempdb.$filename" connection="$local" type="CSV" overwrite="1" delimiter=";" quote='"'/>
        <xdtl:query cmd="SELECT COUNT(*) FROM pg_tables WHERE tablename='$tablename'" target="exist" connection="$local"/>
        <xdtl:if expr="${exist != 0}">
        <xdtl:query cmd="INSERT INTO $tablename
          SELECT t0.* 
          FROM $tempdb.$filename t0
          LEFT OUTER JOIN $tablename t1 ON t1.id = t0.id
          WHERE t1.id IS NULL" connection="$local"/>
        </xdtl:if>
        <xdtl:move source="$tmpdir/$filename.zip" target="$bakdir/$filename.$xdtlDateCode.rar" overwrite="1"/>
        <xdtl:clear target="$tmpdir/$filename.*"/>
      </xdtl:steps>
    </xdtl:task>
  </xdtl:tasks>
</xdtl:package>

In short, a text file is downloaded (get), unzipped (unpack), stripped from CR's with sed (strip) and gets loaded into staging area (read). If the target table exists the rows that are not in the target table already get INSERTed. Finally, incoming file is archived away (move) and the temp files are trashed (clear). All in all, a typical plain vanilla data warehousing task. At this point, some explanations are probably needed.
 
1. The required parameters, filename and tablename get their values in some other (calling) package outside this one. If not, the package get an error.  The rest of the parameters are optional, ie. they might get their values from a calling procedure, if not they fall back to the defaults. 
 
2. The external utilities (get, unpack, strip etc.) get their context in the global configuration file (resources/globals.xml), like this:
 
<entry key="get">wget %target:-O :% %overwrite::-nc% %source%</entry>
 
%target%, %overwrite% and %source% are evaluated to the actual get element attribute values. 
 
3. xdtlDateCode is a variable that was evaluated to the current date in the startup.js script (there can be an arbitrary number of such variable definitions): 
 
var xdtlDateCode = java.lang.String.format("%1$tY%1$tm%1$td", new Array(new java.util.Date()));
 
To be continued...
 
 
 


MMX G4 Design Rationale

July 11, 2010 14:00 by mmx

The primary goals for the next major revision (G4) of MMX Metadata Framework are:

  • OMG MOF compliance, providing facilities to map metamodels created in MOF to MMX Data Model and realize those metamodels in MMX Repository;
  • XML Schema compliance, providing methodology for realization of any metamodel expressed as an XML Schema;
  • support for very basic workflow functionality directly in Core MMX Metamodel;
  • inclusion of Dublin Core attributes and simplified RBAC support directly in MMX Core Metamodel.  

Changes from G3 to G4 in MMX Physical Data Model

Data column Change Comments

md_relation_type.association_type_cd

remove UML association types (association, aggregation, composition) are supported via containment_ind and reliance_ind flags in G4
md_relation_type.semantic_type_cd remove Semantics of a relationship is expressed directly in relationship type in G4
md_relation_type.transitivity_ind
md_relation_type.symmetry_ind
md_relation_type.reflexivity_ind
remove Deprecated. Not supported in G4.
md_relation_type.containment_ind add Indicates that this is a 'contains' association ('aggregation' in UML terms).
md_relation_type.reliance_ind add Indicates that this is an 'owns' or 'relies on' association ('composition' in UML terms). Note that an 'owns' association is also a 'contains' association so to properly indicate an UML composition both flags should contain True. 
md_property_type.domain_cd change Refers to an Enumeration class in metamodel. Replaces domain_type_cd from G3, that combined the meaning of both domain_cd and datatype_cd. 
md_property_type.datatype_cd add Refers to one of the Datatype classes in Core Metamodel. 
md_object.public_ind add Indicates that the object is publicly visible and RBAC permissions for the object are not to be checked. 
md_object.mutable_ind add Indicates that the object can be changed. False would freeze the object. 
md_object.workflow_num add Can be used to tag an object with a workflow state. Note that there are no universal states defined: an application is free to interpret this figure however it chooses to.
md_object.security_id remove Deprecated. Simplified RBAC implementation in G4 does not provide support for RBAC Object concept, G4 RBAC API functions realize this functionality in a more direct way instead. 
md_poperty.domain_id remove Deprecated. In G4 Enumerations are tracked on M2 level.
md_property.value_id add In MOF, a property can be used to identify an object instance. In that case this field would contain object_id of an object. 

Note 1. Enumerations are treated as Core Metamodel classes derived from DataType class and are realized on M2 level. EnumLiterals are realized as instances of one particular Enumeration class and are therefore stored on M1 level, ie. md_object.object_type_cd points to an object_type_cd on M2 level (md_object_type.object_type_cd).

Note 2. Core Metamodel provides very simple facilities to implement basic workflow functionality. These simple facilities come in form of state_ind, public_ind, mutable_ind and workflow_num fields of md_object table that, in combination with published_, edited_, created_ and changed_ timestamps are sufficient to handle basic workflow management needs. The details of a specific workflow implementation are left to an application developers.

Note 3. A simplified, 'low-calorie' RBAC implementation is part of Core Metamodel. User, Role and Permission are implemented as classes of RBAC metamodel. However, RBAC Object is realized as a property of a Permission object. This allows Permissions with multiple Objects, and both class-based (referring a class in M2) and object-based (referring a concrete object in M1) permissions are possible, even in mixed manner. Similarly, RBAC Operation is realized as a property of a Permission object, with enumerated value list stored in M1 as EnumLiterals. Again, interpretation of these Operation tokens is up to an application 'owning' those operations. Finally, in addition to standard RBAC features classes Privilege and Pattern provide additional functionality required to build easy-to-use permission management applications. Privilege acts as a template for Permission objects, Pattern as a template for Role objects.