This chapter introduces the architecture and components of the Media Cataloger. The Cataloger daemon controls processes to create and maintain file metadata. The information in these metadata stores is accessible to applications through the corresponding Media Selector Data Model. In addition, the Cataloger environment provides the alp_catalog (in previous releases, msutil) command-line tool for debugging and testing applications that use Media Cataloger.

Overview ^TOP^

The Media Cataloger controls the processes that collect and maintain file metadata stores. These stores include internal metadata databases that contain information about the location and properties of files available to the user from the device or an external storage card. The Cataloger provides applications access to that information through the public data model. The Cataloger interacts with Extractor libraries, registered through bundle manifests, to extract metadata for supported file types.

The Cataloger is designed for use by applications that manage content; it is not specific to the Media Selector.

Cataloger Architecture and Components ^TOP^

The Cataloger is composed of the elements:

• The mediacatd daemon and its APIs.
• Extractor Host and lookup table, to interact with Extractor Libraries, described in Chapter 9, "Using Extractor Plug-Ins."
• Private metadata stores and their public interface, the Media Selector Data Model.

Cataloger Metadata Storage ^TOP^

The Cataloger stores metadata in internal databases. It provides a public data model for applications to access metadata.

Cataloger Internal Information ^TOP^

A meta database consists of a number of tables that map files, their associated attributes, and physical locations. This includes a database for the device and databases for each attached storage medium.

Meta databases have two types of entities to manage metadata:

• The metadata databases, represented by any number of tables, one for the device and for each card, that contain media types, categories, and descriptive file information, and information about the file's location.
• The cataloger database, stored on the device.
  • The cataloger.db maintains a list of registered device directories (card directories are implicitly registered). The Media Selector does not catalog any file that's not in or under a registered directory.
  • The cataloger.db also maintains a list of extension to MIME and thumbnailer mappings. (Each extension can have several different MIME types, but can have one and only one thumbnailer.)
  • The cataloger.db maintains a list of extensions and extractors. (Each extension may be handled by multiple extractors.) Tree Walker threads check this information so they can quickly process file types. This structure provides lookup functionality to return a handle to an Extractor library, given a file type. Extractors handling multiple extension types can register more than once.Files with unsupported types are listed in the category type OTHER (and the Cataloger stores the filename, size, and modification time), if they aren't in.hidden directories.

Media Selector Data Model ^TOP^

In addition to the tables that it employs to store metadata, the Cataloger maintains a public data model which reflects that store metadata for all available media files. These metadata apis compose the Media Selector Data Model and provide access to metadata for applications that manipulate files.

See "The Media Selector Data Model" for more information about the Media Selector Data Model.

Cataloger Components ^TOP^

The Cataloger consists of a daemon mediacatd and its APIs. The Cataloger API uses the standard IPC to control and query mediacatd. The Cataloger daemon runs as a background process. The system automatically launches the Cataloger at boot time or system reset.

The Cataloger is composed of three major modules, described in the following sections:

• Volume Mounter
• Tree Walker
• File System Monitor

The Tree Walker performs the bulk of the processing, and the other two modules serve as event sources to drive the process.

The Cataloger broadcasts messages as a result of certain operations, for example, when files are deleted, indexed, or unindexed, and sends event notification when scanning directories reading metadata.

Volume Mounter ^TOP^

The Cataloger calls the Volume Mounter every time a new volume is mounted or unmounted (such as when a card is inserted). This module performs the following operations:

• When a volume is mounted, Volume Mounter checks for the following:
  • Is there an existing meta database for the volume?
  • Does the volume have read-only access?

It takes the following actions, based on those conditions:

• If the volume is read-only, and a meta database exists, the Volume Mounter validates the database.
  • If the meta database is valid, the process is complete.
  • If the database is invalid, it creates a temporary meta database in internal memory.
• If the volume is read-only, and a meta database does not exist, the Volume Mounter checks for the presence of an internal temporary meta database.
  • If a temp meta database exists, the Volume Mounter checks if that database was for the volume that was just mounted. If so, it validates that database.
  • If a temp database doesn't exist, or it's not associated with the new volume, the Volume Mounter rebuilds the temp database.
• If the volume is not read-only and a meta database exists, it is validated, otherwise it is created on that volume.
• When a volume is unmounted, the Volume Mounter kills the corresponding Tree Walker thread (if it exists).

Tree Walker ^TOP^

A Tree Walker is a thread spawned in the Cataloger's process that searches a volume for changes to media files and records those changes in the meta database. The Cataloger employs the Tree Walker to efficiently validate a meta database against a volume, making corrections to reflect any changes on the volume. The Cataloger launches Tree Walker threads to make sure that the contents of a meta database are in sync with the contents of a volume.

The Cataloger passes the Tree Walker threads a root path, and a pointer to a list of "forced update extensions". (These are extensions supported by a new (or updated) extractor.) The Tree Walker passes these types of file to the appropriate extractors, even if they are already in the database.

Tree Walkers operating on internal storage are a special case, as they operate on the list of included directories (in the Cataloger's private database) rather than a volume's root node.

In the simple case, where no meta database exists:

• the Tree Walker simply walks the volume, inserting directory path names and "last modified" dates to the directory tables.
• The Tree Walker compares the extension of any file it encounters using a lookup on the Cataloger's data structure.

  If a file extension is unknown, it is trivially rejected.

  If the Tree Walker finds a file with a known file extension, it takes the following actions:

  • It looks up that file type's Extractor library in the Cataloger's data structure.
  • The Tree Walker then passes a reference to the file to that library, and blocks until metadata is formatted and returned.
  • The Tree Walker inserts the metadata to the database.

    Note that this final write needn't block on the database because writes are accumulated for more efficient transaction execution.

    Note that the meta database needs to have a primary key on the media file's pathname + filename —this way inserts fail if the row already exists (for example, when an application manually calls to update the media file's directory just before a Tree Walker encounters and indexes the file).

If the volume passed to the Tree Walker already has a meta database, the Tree Walker validates the contents of the volume to ensure that they still match the contents of the meta database. To do that, it must simultaneously walk the meta database's directory table and the volume's directory tree.

4. The Tree Walker checks each file in the volume against the database to see if it should be added or removed. While iterating the directory list in the meta database, if the corresponding directory doesn't exist on the volume, the Tree Walker removes it from the meta database and checks for media files to delete from that directory.
5. After the Cataloger iterates through the directory list in the meta database, the Tree Walker traverses its list of directories in the volume's directory tree. For each directory, it checks to see if it was in the meta database, and if not, it checks for medias files to add. If they exist, it adds that directory to the directory list

The Tree Walker skips all subdirectories (of a registered directory) whose names start with a dot. This includes . and .., directories the Cataloger creates like .thumbnails and .read_only_media, as well as any third-party configuration directories like .gstreamer or .garnet_app_private.

The Tree Walker directory synchronization algorithm assumes that a the most common mode for Cataloger is to deal with a complex hierarchy with relatively few structural changes each time it's revisited. If that's not the case, and the complex hierarchy is likely to be very different each time the volume is traversed, this algorithm is less efficient than just recreating the meta database from scratch.

Because any volume can suddenly disappear (for example, when a card is ejected), the Tree Walker is sensitive to removable media. It checks error codes after every file access command, and if it encounters any media errors, the thread quits.

The Cataloger can kill a Tree Walker thread at any time that the volume is no longer accessible.

File System Monitor ^TOP^

The Cataloger uses the File System Monitor component to follow events that require changes to the meta databases. Updates take place at the following discreet events:

• The Cataloger received an event from the system, notifying it that a new volume has been mounted or a card inserted.
• The Cataloger received an event from the system, notifying it that a volume has been unmounted or a card removed.
• An application calls the alp_ms_cat_update() API after writing some media data or launching a Media Selector application.
• An application calls the alp_ms_[un]register_directory() API after an application adds or removes a special, non-standard media directory (designated to monitor).
• An application calls the alp_ms_[un]index_file() API after an application adds or removes a single media file in a monitored registered media directory.

The Media Selector Data Model ^TOP^

The Media Selector Data Model provides a mechanism through which applications can access to the Cataloger's file metadata. The Extractors, when they extract metadata from media files, and applications, when they create media files, use the Data Model API to load values into tables in the internal databases. Applications can then use the Data Model API to query the public data model for information about those files. The Media Selector GUI presents that information to users.

The Media Selector Data Model Query ^TOP^

To search the metadata, the Media Selector Service supports these basic features:

• Category selection (All, Audio,...) Once a category is chosen, submenus of Media Selector allow users to specify values for filters and sorts.
• Filter (All, Games, Utilities, Favorites, lists of included or excluded categories,...)
• Sorts
  • Sort by (Data Size, File Name, File Type, modification time, location) In some cases, there may be multiple sort keys, as in Media Selector for audio:
  • Sort order (ascending or descending)

For more information about these features, see "Searching with Media Selector.".

The Cataloger provides the alp_ms_dml_statement_parameters structure as an area in which to construct queries; with this an application can set values to create queries to the public information in the data model. Use the alp_ms_dml_statement_parameters_* functions (described in Chapter 5, "Introducing Media Cataloger,") to set the following fields to build your application's query:

Table 5.1  DML Query Statement Parameters

Media Categories and Their Contents ^TOP^

The Cataloger implements category selection by maintaining one table per category. Choosing a category determines which meta databases to use with the data model. Each category, and its corresponding item type, has a standard set of column headers defined in the data model, based on the type of information that meta database contains.

The ALP_MS_CATEGORY_ALL selection includes all file types.

Applications can create a special category, ALP_MS_CATEGORY_COLLECTION, to group files of similar type.

• ALP_MS_DML_COL_COLLECTION_TITLE describes a string value of the collection title,
• ALP_MS_DML_COL_COLLECTION_COUNT, an int32 value of the number of items in the collection.

If your file type needs to store additional metadata, or you want to store additional metadata in standard categories, you can extend the metadata schema for your type by including additional tags in your extractor manifest. You then use the Media Selector DML API to put data into your custom categories and custom columns in exactly the same way that you populate factory-standard data fields. See"Category Extension" for more information.

Available contents, by standard category, are as follows:

Table 5.2  Media Selector Data Model Columns

The alp_catalog Tool ^TOP^

The Media Selector utility, alp_catalog (in previous releases, mustil), is a command line program which allows you to interact directly with the Media Selector subsystem to interrogate the system and modify values. The functionality provided by this tool is available to applications through the Media Cataloger APIs. This tool helps you debug as you develop applications that use the Media Selector Service. Use this command to directly examine the contents of the meta databases and verify the available extractor plug-ins.

Syntax and Options ^TOP^

alp_catalog -[efluUx]

alp_catalog -[diIru] path

where

path specifies the path to the directory or file.

Table 5.3