Kamaelia.Support.Data.Repository

• Kamaelia component repository introspection
  • Example Usage
    • Simple lists of component/prefab names
    • Detailed introspections::
  • Obtaining introspection data
  • How are components and prefabs detected?
  • Structure of detailed introspections
  • Implementation Details

Kamaelia component repository introspection

This support code scans through a Kamaelia installation detecting components and picking up relevant information such as doc strings, initializer arguments and the declared Inboxes and Outboxes.

It not only detects components and prefabs, but also picks up modules, classes and functions - making this a good source for documentation generation.

Example Usage

Simple lists of component/prefab names

Fetch a flat listing of all components. The key is the module path (as a tuple) and the value is a list of the names of the components found:

>>> r=Repository.GetAllKamaeliaComponents()
>>> r[('Kamaelia','Util','Console')]
['ConsoleEchoer', 'ConsoleReader']

Fetch a nested listing of all components. The leaf is a list of entity names:

>>> r=Repository.GetAllKamaeliaComponentsNested()
>>> r['Kamaelia']['Util']['Console']
['ConsoleEchoer', 'ConsoleReader']

Fetch a flat listing of all prefabs:

>>> p=Repository.GetAllKamaeliaPrefabs()
>>> p[('Kamaelia','File','Reading')]
['RateControlledFileReader', 'RateControlledReusableFileReader',
'ReusableFileReader', 'FixedRateControlledReusableFileReader']

Fetch a nested listing of all prefabs:

>>> p=Repository.GetAllKamaeliaPrefabsNested()
>>> p['Kamaelia']['File']['Reading']
['RateControlledFileReader', 'RateControlledReusableFileReader',
'ReusableFileReader', 'FixedRateControlledReusableFileReader']

Fetching a flat listing of components as defined in a specific directory (rather than the current Kamaelia installation):

>>> r=Repository.GetAllKamaeliaComponents(baseDir="/data/my-projects/my-components/")

Detailed introspections::

We can ask for a complete introspection of the current Kamaelia installation:

>>> docTree=Repository.ModuleDoc("Kamaelia","/usr/lib/python/site-packages/Kamaelia")
>>> docTree.resolve(roots={"Kamaelia":docTree})

And look up a particular module:

 >>> m=docTree.find("Util.Console")
 >>> m
<Repository.ModuleDoc object at 0x40403b0c>

Then find components declared in that module:

>>> cs=m.listAllComponents()
>>> cs
[('ConsoleReader', <Repository.ClassScope object at 0x41511bac>), ('ConsoleEchoer', <Repository.ClassScope object at 0x4115990c>)]
>>> (name,c)=cs[0]
>>> name
'ConsoleReader'
>>> c
<Repository.ClassScope object at 0x41511bac>

And look at properties of that component:

>>> c.module
'Kamaelia.Util.Console'
>>> c.inboxes
{'control': 'NOT USED', 'inbox': 'NOT USED'}
>>> c.outboxes
{'outbox': 'Lines that were typed at the console', 'signal': 'NOT USED'}
>>> print c.doc
ConsoleReader([prompt][,eol]) -> new ConsoleReader component.

Component that provides a console for typing in stuff. Each line is output
from the "outbox" outbox one at a time.

Keyword arguments:

- prompt  -- Command prompt (default=">>> ")
- eol     -- End of line character(s) to put on end of every line outputted (default is newline)

This includes methods defined in it:

>>> c.listAllFunctions()
[('main', <Repository.FunctionScope object at 0x4166822c>), ('__init__', <Repository.FunctionScope object at 0x4166224c>)]
>>> name,f=c.listAllFunctions()[1]
>>> name
'__init__'
>>> f
<Repository.FunctionScope object at 0x4166224c>

We can look at the docs for the function:

>>> f.doc
'x.__init__(...) initializes x; see x.__class__.__doc__ for signature'

We can ask for a string summarising the method's arguments:

>>> f.argString
'self[, prompt][, eol]'

Or a list naming each argument, consisting of (argname, summary-representation) pairs:

>>> f.args
[('self', 'self'), ('prompt', '[prompt]'), ('eol', '[eol]')]

Obtaining introspection data

To get a detailed introspection you create a ModuleDoc object. You can either point it at a specific directory, or just let it introspect the currently installed Kamaelia repository.

You can specify the module path corresponding to that directory (the "root name"). The default is simply "Kamaelia". If for example, you point it at the Kamaelia.Chassis directory; you should explain that the root name is "Kamaelia.Chassis". Or if, for example, you are using this code to document Axon, you would specify a root name of "Axon".

After instantiating your ModuleDoc object; remember to call its "resolve" method to allow it to resolve references to base classes, and determine method the resolution order for classes.

How are components and prefabs detected?

Components and prefabs are detected in sourcefiles by looking for declarations of an __kamaelia_components__ and __kamaelia_prefabs__ variables, for example:

__kamaelia_components__ = [ "IcecastClient", "IcecastDemux", "IcecastStreamWriter" ]
__kamaelia_prefabs__ = [ "IcecastStreamRemoveMetadata" ]

They should be declared individually, at module level, and should consist of a simple list of strings giving the names of the components/prefabs present.

Structure of detailed introspections

The introspection is a hierarchy of Scope objects, each representing a delcaration scope - such as a module, class, function, etc. These are built up to reflect the structure of the library if it is imported.

• ModuleDoc objects represent each module. They may contain:

  • Other ModuleDoc objects
  • ImportScope objects
  • ClassScope objects (representing classes and components)
  • FunctionScope objects (repesenting functions and prefabs)
  • UnparsedScope objects (anything that wasn't parsed)

ClassScope and FunctionScope objects may also contain any of these. For example, methods in a class will be represented as FunctionScope objects within the ClassScope object.

The find() method of any given scope can be used to lookup a symbol in that scope, or its children. For example, you could call find() on the "Kamaelia.Chassis" ModuleDoc object with the argument "Graphline.Graphline" to retrieve the graphline component (its full path is "Kamaelia.Chassis.Graphline.Graphline")

The listAllXXXXX() methods enumerate items - such as classes, functions, components, prefabs or modules.

Implementation Details

This code uses the python compiler.ast module to parse the source of python files, rather than import them. This allows introspection of code that might not necessarily run on the system at hand - perhaps because not all dependancies can be satisfied.

Basic tracking of assignment operations is performed, so the following is fair game:

from Axon.Component import component as flurble

class Boo(flurble):
    pass

Foo=Boo

However anything more comple is not processed. For example, functions and classes declared within "if" statement will not be found:

if 1:
    class WillNotBeDetected:
        pass

    def AlsoWillNotBeDetected():
        pass

The simplified functions that only return lists of component/prefab names ( GetAllKamaeliaComponentsNested, GetAllKamaeliaComponents, GetAllKamaeliaPrefabsNested and GetAllKamaeliaPrefabs) simply run the full introspection of the codebase but then throw most of the information away.

---