Abstractions from Multimedia Hardware Abstraction

Transcription

Lehrstuhl für Informatik 4
Kommunikation und verteilte Systeme
Abstractions from Multimedia Hardware
State of the art of programming
• Application code is highly dependent on hardware
• Change of multimedia devices still often requires re-implementation
• Example: sequence of (very) specific instructions:
Chapter 2: Basics
Chapter 3: Multimedia Systems – Communication Aspects and Services
Chapter 4: Multimedia Systems – Storage Aspects
Chapter 5: Multimedia Usage and Applications
• Documents and Hypermedia
• Multimedia User Interfaces
• Abstractions for Programming
• Multimedia Applications
5.3: Abstractions for Programming
• Abstraction Levels
• System Software and
Libraries
• Frameworks
• Higher Programming
Languages
• Object-Oriented Approaches
Page 1
Chapter 5.3: Abstractions for Programming
static unsigned long read_timer(void)
{
/* example taken from Linux kernel code - i386 architecture */
unsigned long t, flags;
int i;
__save_flags(flags);
__cli();
t = jiffies * 11932;
outb_p(0, 0x43);
Strong hardware dependency may cause
i = inb_p(0x40);
problems with:
i |= inb(0x40) << 8;
• Portability
__restore_flags(flags);
• Reusability
return (t - i);
}
• Coding efficiency
Page 2
Abstraction Levels
Libraries
• Common operating system extensions try to solve this problem
• Different programming possibilities for accessing and representing multimedia data:
• Processing of continuous media based on functions embedded in libraries
• Libraries differ in their degree of abstraction
• Example from IBM’s early Audio Visual Connection (AVC):
Multimedia application
Object-oriented
programming
languages
acb.channel = AAPI_CHNA
acb.mode = AAPI_PLAY
...
aud_init(&acb) /* acb is the audio control block */
...
audrc = fab_open(AudioFullFileName,AAFB_OPEN,AAFB_EXNO, 0,&fab,0,0,0,0);
fork(START in PARALLEL)
aud_strt(&acb)
displayPosition(RelativeStarttime, Duration)
...
Higher
programming
languages
Frameworks
System software
Libraries
Device drivers for continuous media
Device
Page 3
Page 4
Libraries - OpenGL
System Software
2D and 3D graphics API developed by Silicon Graphics
Device access becomes part of the operating system:
• Basic idea: “write applications once, deploy across many platforms”:
PCs
Workstations
Super Computers
• Data as time capsules (file extensions)
Each Logical Data Unit (LDU) carries in its time capsule its data type, actual
value and valid life span
Useful concept for video, where each frame has a valid life span of 40ms (rate
of read access during a normal presentation)
Presentation rate is changed for VCR (Video Cassette Recorder) functions like
fast forward, slow forward or fast rewind by
• Changing the presentation life span of a LDU
• Skipping of LDUs or repetition of LDUs
• Benefits:
Stable
Reliable and Portable
Evolving
Scalable (Features like Zoom, Rectangle handling ...)
Well documented and easy to use
• Data as streams
A stream denotes the continuous flow of audio and video data between a
source and a sink
Prior to the flow the stream is established equivalent to the setup of a
connection in a networked environment
• Integrated with:
Windows 95/NT/2000/XP
UNIX X Window System
Page 5
Page 6
System Software - MCI
System Software - DirectX
Low-level APIs and libraries for high-performance applications
• Especially games - formerly known as the "Game SDK"
Windows Media Control Interface (MCI):
Direct access to hardware services
• E.g. audio & video cards, hardware accelerators
• “DirectX” = “direct access”
• Strong relationship/interaction with ActiveX/DCOM
MMSYSTEM library for
extensibility and device
independence
Page 7
Page 8
System Software - DirectX
Frameworks
Additional level of abstraction:
Components:
• DirectDraw - 2 dimensional graphics capabilities
• Direct3D - extensively functional 3D graphics programming API
• DirectSound - (3D) sound, mixing and playback of multiple streams
• DirectPlay - for network multiplayer game development
• DirectInput - input from various peripherals, e.g. joysticks, data gloves
• Combination of different media
• New data types
E.g. mixed.video := orig.video + subtitle
Granularity problematic (pixel, frame, image group, sequence, video, ...)
• Reuse of modules:
Synchronization
Scaling
(De)compression
File formats
…
Implementation Strategy:
• Hardware Abstraction Layer (HAL)
• Hardware Emulation Layer (HEL)
• Media Layer (for aggregated “high level” functionality)
Animations
Media streaming
Synchronization
Application frame given: generic problem solution
Creation of specialized application components: media containers, media processors
Page 9
Frameworks: JMF
Higher Programming Languages
Media as data types:
• Definition of appropriate data types (e.g. for video and audio)
• Smallest unit can be a LDU
• Example of merging a text and a motion picture:
Framework controls execution
• E.g. myApplicationComponent.init(); myApplication.start();
• Example: Java Media Framework (JMF)
JMF supports platform-neutral synchronized playback of multimedia data
• Local or via network:
Pull protocols - e.g. HTTP
Push protocols - e.g. RTP
Easy integration into platforms native environment and core packages; supported media
formats:
• MPEG, QuickTime, AVI
• WAV, AU, MIDI
Extensible:
• New Data Sources (e.g. for FTP or Video on Demand (VOD) protocols)
• New Players
Page 10
Page 11
subtitle TEXT_STRING
mixed.video, ldu.video VIDEO_LDU;
...
WHILE
COBEGIN
PROCESS_1
input(av_filehandle,ldu.video)
IF new_video_scene
input(subtitle_filehandle,subtitle)
mixed.video := ldu.video + subtitle
PROCESS_2
output(video_window,mixed.video)
...
END_WHILE
...
In the above example, implicit type conversion must occur
Real-time processing of LDUs depend on their granularity and can be problematic in a
high-level
language for Programming
Page 12
Chapter
5.3: Abstractions
Media as files: instead of considering continuous media as data types they can be
considered as files:
Media as processes: it is possible to map continuous media to processes and
integrate them into a high-level language:
file_h1 = open(MICROPHONE_1,...)
file_h2 = open(MICROPHONE_2,...)
PROCESS cont_process_a;
...
on_message_do
set_volume ...
set_loudness ...
...
...
[main]
pid = create(cont_process_a)
send(pid, set_volume, 3)
send(pid, set_loudness)
...
file_h3 = open(SPEAKER, ...)
...
read(file_h1)
read(file_h2)
mix(file_h3, file_h1, file_h2)
activate(file_h1, file_h2, file_h3)
...
deactivate(file_h1, file_h2, file_h3)
...
rc1 = close(file_h1)
This process implements a set of actions
(“set-volume”, “set-loudness”)
Page 13
Page 14
Programming Language Requirements
Object-Oriented Approaches
The high-level language should support parallel processing, because the processing
of continuous data is
• controlled by the language through pure asynchronous instructions
• an integral part of a program through the identification of media
Different processes must be able to communicate through an inter-process
communication mechanism, which must be able to:
• Understand a priori and/or implicitly specified time requirements (QoS parameters or
extracted from the data type)
• Transmit the continuous data according to the requirements
• Initiate the processing of the received continuous process on time
Page 15
Basic ideas of object-oriented programming is data encapsulation in connection with
class and object definitions
• Abstract Type Definition (definition of data types through abstract interfaces)
• Class (implementation of a abstract data type)
• Object (instance of a class)
Other important properties of object-oriented systems are:
• Inheritance
• Polymorphism
Page 16
Object-oriented Approaches
Devices as classes: devices are assigned to objects which represent their behavior
and interface
Processing units as classes:
• Three main objects:
Source objects
Destination objects
Combined source-destination objects allows the creation of data flow paths
through connection of objects
class media_device {
char *name;
public:
void on(), off();
};
class media_in_device:
public media_device {
private:
DATA data;
public:
refDATA get_data();
};
• Multimedia object
Basic Multimedia Classes (BMCs) /
Basic Multimedia Objects (BMOs)
Compound Multimedia Classes (CMCs) /
Compound Multimedia Objects (CMO), which are compound of BMCs / BMOs
and other CMCs/CMOs
BMOs and CMOs can be distributed over different computer nodes
class media_out_device:
public media_device {
public:
void put_data(refDATA dat);
};
Page 17
Object-oriented Approaches
Media as classes:
• Media Class Hierarchies define hierarchical relations for different media
• Different class hierarchies are better suited for different applications
Example of CMC (simplified notation):
Lexicon: compound_class;
DATA: Explain external;
Animation external;
Example:
Medium
Acoustic_Medium
Music
Opus
Score
Audio_Block
Sample_Value
Speech
...
...
Opitcal_Medium
Video
Video_Scene
ACCESS_POINTS: VIDEO_SOURCE Animation.VIDEO_SOURCE;
AUDIO_SOURCE Animation.AUDIO_SOURCE;
TEXT_SOURCE Explain.TEXT_SOURCE;
METHODS:
start:
play:
pause:
stop:
...
display(Explain);
play(Animation);
pause(Animation);
stop(Animation);
Page 18
Page 19
Video
Video_Scene
Image
Image_Segment
Pixel
Line
Pixel
Column
Pixel
Animation
...
Text
...
Page 20

Abstractions from Multimedia Hardware Abstraction

Transcription

Documents pareils

Firewall Tasks of a Firewall Tasks of a Firewall

Database Systems Multimedia Database Management

seminar invitation - Lehrstuhl für Systemdynamik und Prozessführung

Network Infrastructure Infrastructure Components: Hub

Secure Socket Layer (SSL) Handshake Protocol

Glass pipes for sale - Hookah Online Store

Attack - Informatik 4

Synchronization in Distributed Systems

Chapter 2 - Informatik 4

Die Children`s improvement organization setzt sich für Waisen und

Polina Kondratieva - Fakultät für Informatik

Basis of Current Mobile Systems

Metropolitan Area Networks

• Distributed object-based system • Common Object Request Broker

4.2: Kerberos • Kerberos V4 • Kerberos V5 Chapter 2: Security

03 Push