The Multimedia Framework's media engine encapsulates the GStreamer core and plugins. It drives the application requests for audio or video playback through the use of media sessions. Each media session represents a recording or playback request; the session controls the data transport for that request.
To use the Media Framework for playback, for example, you create a session, tell it where to find the data to be played, and then tell the session to start playing. The session automatically calls the appropriate GStreamer decoders, and sends the decoded data to the appropriate output device. Video is sent to the screen, whereas sound is first routed through the Audio Manager, which streams it to an output device such as a speaker.
Components of a Media Session
A media session is created by an application wishing to process some form of media in a format such as MPEG-4 or WAV (the formats supported on a given device depend upon the installed codecs). A session is made up of a source object, one or more destination objects, and the appropriate GStreamer plugins.
Source object
A source object is a container for a source device. For a recording session, the source device might be a camera or a microphone. For a playback session, the source device might be a file or a network stream. In both cases the source object is responsible for processing incoming data from the device or file.
Destination object
A destination, or sink object is the container for the destination device. For a recording session, the destination device might be a file or network stream. For a playback session, the destination might be the screen and/or the speakers. In both cases the destination object is responsible for streaming the data to the output devices.
Note that a media session can have multiple destination devices. For instance, an audio/video stream can be sent to both the display and a speaker, allowing the stream to be both seen and heard.
GStreamer
GStreamer is an open-source media solution that uses a plugin architecture. Plugin libraries are dynamically loaded to support a wide spectrum of codecs, container formats and input/output drivers. In ACCESS Linux Platform, the appropriate GStreamer plugin is loaded by the source object to decipher the data that is then piped to the destination object.
Supported Codecs
In order to play from and capture to encoded media files, the media framework uses software components called codecs. A codec translates media data from one format to another. Each codec supports a particular encoding algorithm, such as MP3 or MS-ADPCM. Note that although GStreamer is most commonly though of as processing time-based media (audio, video), with the proper codecs it can process static data—such as a JPEG or GIF image—as well.
Table 2.1 lists a number of common codecs found on various ACCESS Linux Platform devices. File formats in this table are distinct from codecs due to the fact that a file format may encapsulate many kinds of encoded data.
IMPORTANT: The set of codecs installed by default on any given device is up to the device manufacturer, and, if the manufacturer allows it, the end user may be able to install additional codecs after-the-fact. Thus, Table 2.1 is intended to serve as a representative set of codecs and should not be taken as definitive.
A media session uses a decoder during playback. During recording, the session uses an encoder before writing to the file stream.
Types of Media Sessions
There are two types of media sessions:
These two session types cover the following usage scenarios:
Later chapters in this book detail the precise steps involved in implementing each of these scenarios.
Workflow
Figure 2.1 shows the calls that are made and the objects that are involved when playing back media stored in a file. The architecture is similar for other media session types.
Figure 2.1 Playback walkthrough
The rest of this chapter walks you through the steps of creating a media session with the API set. Refer to the Multimedia module in the API reference as you read through the steps in the next few sections.
Media Session Headers and Libraries
The multimedia library, libalp_media.so, implements the application-level APIs used to control the playback and recording of audio-visual media. These APIs provide a standard means for applications to reference media content stored locally on the device, stored on a network, or accessible from some attached hardware device such as a microphone or camera. This library should be included in your makefile (add alp_media to the list of libraries in your makefile), along with the rest of the libraries you will be linking against.
The header files used with multimedia include the following:
-
media_session.h - Contains more of the structures and constants related to a media session. In addition, this file contains the API declarations that control session management.
-
media_defs.h - Contains some of the structures and constants related to a media session. This header file is included by
media_session.h; you don't need to include it in your application.
To use these files add the following line to your code:
#include <alp/media_session.h>
The Structure of a Multimedia Application
ACCESS Linux Platform applications, whether they play audio and/or video, record audio and/or video, or capture still images, all use the same basic steps when interacting with the media engine. Generally, they:
- Create a Media Session
- Add Source Objects
- Add Destination Objects
- Set Session Properties
- Register a Callback (Optional)
- Finalize the Session and Initiate the Action
Each of these steps is discussed in some detail in the following sections. Then, the next couple of chapters walk through the precise steps needed to play, record, and capture.
Create a Media Session
To create a media session the first call you must make is to alp_media_session_create(). This creates a context in which the media is processed. Note that a session is created for a specific "action," either playback or record. A session is created for playback by specifying ALP_MM_SESSION_CLASS_PLAYBACK in the SessionClass parameter. To record, create a session with the sessionClass parameter set to ALP_MM_SESSION_CLASS_RECORD. The alp_media_session_create() function returns a pointer to an opaque structure representing the new media session in the SessionID parameter; you pass this session pointer in subsequent media session function calls that interact with that session.
Add Source Objects
When you add a source object, the media session opens the source of the data, determines the format, and loads the appropriate codecs needed to decode the information. You add a source object with a call to alp_media_add_source(). The first parameter is the sessionID returned when you opened the media session. The second parameter, AlpMMSourceType, defines where the data is coming from. For a playback session the source will be one of the following:
- ALP_MM_TYPE_FILE
- A local file.
- ALP_MM_TYPE_FILELIST
- A list of files (that is, a playlist).
- ALP_MM_TYPE_NETSTREAM
- A network stream.
- ALP_MM_TYPE_MEMSTREAM
- A pointer to the beginning of a stream in memory.
For a recording session, the source can either be:
The sourceURL parameter specifies the path to the source. For the source device types (microphone and camera), constant sourceURL paths are defined in alp/media_session.h: DSPDEV for the microphone device, and CAMERADEV for the camera device.
Note that for a recording session you can add more than one source device. For instance, you can add both the microphone and camera devices to record both video and audio simultaneously. Add each source using a separate alp_media_add_source() call.
Add Destination Objects
To add a destination object, you call alp_media_session_add_dest() and specify a valid destination. One of the following:
-
DSPDEV("dev://dsp") - the current audio output device (speaker, headset, etc.) -
FILENAME("dev://filename") - a file -
LCDDEV("dev://lcd") - the display
If you have multiple streams—typically, audio and video—specify multiple destinations by calling this function multiple times. For instance:
alp_media_session_add_dest(session_id, LCDDEV, &lcd_dest_id); alp_media_session_add_dest(session_id, DSPDEV, &dsp_dest_id);
When specifying a file as the destination, you need to then provide the actual path to the file by setting the ALP_MM_DESTINATION_URL property, like this:
int size = 0; char *filename = NULL; filename = (char *)malloc(128 * sizeof(char)); strcpy (filename, "/tmp/media/wav.wav"); alp_media_property_set(g_idsession, ALP_MM_DESTINATION_URL, ALP_MM_PROPERTY_CODE_STRING, (void *)filename, (void *)&size);
Note that the supplied path string is NULL-terminated. Thus, although you must supply a non-NULL pointer for the size parameter, the value to which it points needn't reflect the actual length of the string.
Set Session Properties
There are a great many session properties you can set to tailor the recording or playback session, such as the type of codec to use, the size and position of the rectangle on the display to use to present the recorded video stream, or various camera settings. See "Media Session Properties - Record and Capture" and "Media Session Properties - Playback" for descriptions of the various properties.
Settings can be adjusted both before and after you finalize the session.
Register a Callback (Optional)
If your application will benefit from being notified when errors and warnings arise, when the session state changes (from playback to stopped, for instance), or on a periodic basis during playback, write an event handler and register it as a media session callback function. See "Registering a Callback" for details.
Finalize the Session and Initiate the Action
Once you have set the sources and destinations required for your session, before you actually begin the recording or playback you must finalize the session. This is simply a matter of calling alp_media_session_finalize(). Then, you issue "control" commands by supplying one of the AlpMMSessionControlOpcode enum values to alp_media_session_control().
The set of control commands allow you to play, record, pause, and stop. As well, there are commands to grab a still image, skip to a different media file in a playback list, enable a preview, refresh the display, and prefetch source data. See the API reference documentation for the complete set of commands.
NOTE: How you initiate a particular recording session varies slightly depending upon whether you are recording audio, video, or both. See Chapter 5, "Recording and Capturing," for full details.
Registering a Callback
Each media session can have a single registered callback function that is invoked at various times: when an error or warning is generated, when the session state changes, when the session is deleted, or, during playback, at regular intervals (this last is useful for updating a playback progress indicator).
When your callback function is invoked, it is passed an AlpMMSessionEvent structure; this structure is defined as follows:
typedef struct _AlpMMSessionEvent {
AlpMMSessionID session_ref;
AlpMMEvent event_code;
int32_t event_cause;
int64_t position;
int64_t duration;
int64_t total_estimated_time;
int64_t total_recorded_size;
} AlpMMSessionEvent;
The event_code field is key: it tells you why the callback was invoked. Because you can only have a single callback for a given session, your callback function can key off this value and react accordingly based upon the reason for the callback. The following are the event codes you can expect to receive in your callback:
- ALP_MM_SESSION_EVENT_STATE_CHANGED
- The session's state has changed, for instance from running to stopped.
- ALP_MM_SESSION_EVENT_WARNING
- A recoverable error has occurred.
- ALP_MM_SESSION_EVENT_ERROR
- A fatal error has occurred.
- ALP_MM_SESSION_EVENT_CURRENT_TIME
- During playback, your callback is invoked approximately once per second with this code. The duration and position fields in the
AlpMMSessionEventstructure indicate the progress of the playback session. Note that this event is not sent when recording. - ALP_MM_SESSION_EVENT_DELETING
- The session is in the process of being deleted.
Although the event_cause field has a number of possible values (as defined by the AlpMMSessionEventCauses enum), at this time only two causes are actually produced:
- ALP_MM_SESSION_EVENT_CAUSE_END_OF_STREAM
- The end of the playback stream or recording has been reached.
- ALP_MM_SESSION_EVENT_CAUSE_REQUESTED_BY_APP
- This is the cause ascribed to all other events.
The remaining fields in the AlpMMSessionEvent structure are defined as follows:
-
session_ref - Identifies the media session.
-
position - Indicates the amount of time that has passed since the beginning of a playback session. This field is not defined for recording sessions.
-
duration - The length of the clip being played back. This field is not defined for recording sessions.
-
total_estimated_time - Defined for recording sessions only, this contains an estimate of the remaining recording time.
-
total_recorded_size - Defined for recording sessions only, this field contains the total size of the recording.
Use of a callback function is optional. To register your callback, simply call alp_media_session_register_callback() and pass it a pointer to your callback function. Your callback function must have a prototype that conforms to AlpMMSessionCallbackFn:
typedef void (* AlpMMSessionCallbackFn)(const AlpMMSessionEvent *event,void *userdata)
Note that both the callback registration function and the callback itself have a userdata parameter. This is an optional pointer (pass NULL if you don't want to use it) that you can pass from the callback registration point into the callback function itself. This parameter is simply passed on each time the callback is invoked, and is otherwise not used in any way by the media session. Use this parameter in any way that is appropriate for your application. For instance you could use it to pass the name of the media file you are playing to the callback function, allowing the callback to do something with the file when a given event occurs.
Managing Media Session Volume Levels
For a media session, the volume of the sound that an end user hears is a combination of:
- The session's volume level
- The master volume level
- The volume level set for the output device (such as the speaker)
Because the latter two volume levels affect sounds other than just that being produced by your media session, you should avoid changing the volume level for a given output device as well as the master volume level.
Session Volume Levels
The Media Engine relies upon the Audio Manager to process audio data. Audio is played through a stream of the "audio_out" stream class. Accordingly, you can control your session's volume level by altering the level of the "audio_out" stream class, as described in "Stream Class Volume."
Master Volume Levels
As illustrated in Figure 3.2, the master volume is a "global" volume control that affects all streams simultaneously. Because the master volume affects such a wide range of sound sources—from ring tones to alarms to multimedia audio—applications should avoid altering it. Many devices will have dedicated volume keys that allow the user to raise or lower the master volume level.
If, in the unlikely event you need to affect the master volume level, you would do so as described in "Master Volume.".
Device Volume Levels
The volume level set for a given output device controls all output to that device: all sessions and streams that send audio to that device have the device volume level applied. That is, raising or lowering the device volume level proportionally increases or decreases the volume level of all sources that are sending audio to that device; their volume levels relative to each other (set individually for each source) remain the same.
The Audio Manager has a provision for setting the volume level of each separate output (or input) device independently. However, applications cannot count on a particular stream always being routed to or from a particular device, since as conditions change (for instance, when the user plugs in a headset or when a phone call is received) the audio router may redirect various streams depending upon the rules set forth in the routing table. The device volume levels are best left to the system, allowing the user to, for instance, adjust the relative volume of the speaker and headphones so that listening levels are equally comfortable through either.
