key: cord-0804319-9qvd72aa authors: Tang, Chunxu; Wang, Beinan; Chen, C. Y. Roger; Wu, Huijun title: CWcollab: A Context-Aware Web-Based Collaborative Multimedia System date: 2022-04-13 journal: 2021 IEEE International Conference on Communications, ICC 2021 DOI: 10.1109/icc42927.2021.9500377 sha: e1943c7b00fba380ff14b01ef17aac921277ce43 doc_id: 804319 cord_uid: 9qvd72aa Remote collaboration tools for conferencing and presentation are gaining significant popularity during the COVID-19 pandemic period. Most prior work has issues, such as a) limited support for media types, b) lack of interactivity, for example, an efficient replay mechanism, c) large bandwidth consumption for screen sharing tools. In this paper, we propose a general-purpose multimedia collaboration platform-CWcollab. It supports collaboration on general multimedia by using simple messages to represent media controls with an object-prioritized synchronization approach. Thus, CWcollab can not only support fine-grained accurate collaboration, but also rich functionalities such as replay of these collaboration events. The evaluation shows hundreds of kilobytes can be enough to store the events in a collaboration session for accurate replays, compared with hundreds of megabytes of Google Hangouts. Real-Time Collaboration (RTC) has a long history in both research and engineering domains. It is gaining significant popularity during the COVID-19 pandemic period. For geographically dispersed teams, it is usually essential for teammates to have a remote collaboration tool for conferencing and presentation purposes. Additionally, considering the distributed and rapidly increasing volume of data and expeditious development of modern web browsers, it is necessary to provide an organized web-based group-aware platform to support collaboration among a large number of users. In RTC platforms, not only are the artifacts of multimedia shared but also the controls on the artifacts are broadcast and synchronized. Some of the pioneering work, such as GroupSketch [1] , VideoWhiteboard [2] , and Liveboard [3] , implemented collaborative multimedia systems by capturing users' drawings and projecting them to remote screens. Based on the same technology, various screen sharing products, such as Zoom, Cisco WebEx, and Google Hangouts/Meet, were developed and are now widely used. Recently, with the rapid development of modern web technologies, lots of web-based collaboration tools, including Collabode [4] , RichReview++ [5] , and Tele-Board [6] , were developed. * The work was done when the authors were at Syracuse University, Syracuse, USA. To our knowledge, even though various web-based groupware tools have been developed for versatile purposes such as whiteboard drawing and document editing, no systems can integrate these functionalities to suit general-purpose multimedia collaboration. Kim et al. [7] made a valuable contribution in this direction that they proposed an MVC architecture for ubiquitous collaboration. Their tool still only handled static media like whiteboard drawings and images, without effectively working on other types of media with dynamic contents such as videos and web pages. Furthermore, for current popular screen sharing products, in a specific session, both the contents of the media and manipulations on the media are transmitted by capturing the display continuously, leading to large consumption of network bandwidth. This poses challenges for large geographically dispersed teams with the unstable network quality. By contrast, we propose to split the contents of a presentation into static media resources and dynamic actions. The static media resources, for example, a video or a PDF document, can be transmitted to attendees beforehand; and the dynamic events occurring in a session such as muting a video are broadcast and synchronized on the fly. As a result, a collaboration session is organized as the combination of static materials and dynamic events encapsulated in an event-driven stream of messages. With these messages, we also implemented the precise recording and replay of collaboration events, differentiating our work from traditional collaboration platforms. To cover the research gaps in prior work mentioned above, we propose a context-aware web-based collaborative multimedia system-CWcollab. Specifically, our contributions are: 1) A general-purpose collaborative multimedia system. • Support for general multimedia. Not only static media (PDF documents, images, etc.) but also dynamic media (videos, web pages, etc.), is supported in CWcollab. This notable feature differentiates our work from prior work, as illustrated in Table I . • Support for general events. All actions in a session are captured as events, sent and handled on the fly. This also indicates the system is open to other possible extensions in a plugin pattern. A developer can add supports for various events of collaboration on different kinds of media following our uniform interface, demonstrated in Section III-A. • Support for general environment. Our system is totally web-based. Users can access the system from various platforms, including desktops, tablets, and mobile devices, as long as web browsers are supported. This significantly eliminates the complexity of setup on different platforms. 2) An object-prioritized context-aware approach to capture and replay media actions for rich functionalities with low network bandwidth. • Object-prioritized media controls. To our knowledge, most web-based collaboration tools are position-based or proportion-based, which implies that media controls are synchronized through absolute or relative positions. However, many websites have applied the responsive web design, where the user interface is automatically adjusted on various devices with various screen sizes. This poses challenges for capturing and replaying events with the traditional position-based approach. By contrast, in CWcollab, media controls are related to media objects. We propose an object-prioritized hybrid synchronization approach, representing each action in simple messages, discussed in Section III-B. • Rich functionalities. As each media control is represented by a simple message, CWcollab also supports rich interactive functionalities for collaboration and presentation such as material preparation, real-time synchronization, and precise replay of a session. • Low network bandwidth. The usage of an objectprioritized approach also implies that CWcollab has a very low bandwidth usage, compared with current video conferencing products such as Zoom and Google Hangouts. In our study, hundreds of kilobytes can be enough to store the events in a session, compared with hundreds of megabytes used in screen sharing tools. The remainder of this paper introduces the related work in Section II, discusses the architectural design and implementation in Section III, and evaluates our platform including a comparison with Google Hangouts in Section IV. Section V concludes the paper. RTC is usually considered as a subdomain of Computer Supported Cooperative Work. Previously, people achieved basic remote collaboration by means of video/audio calls. While it is obvious that this form of communication can only allow very preliminary cooperation. Afterward, researchers developed more complicated collaborative applications. Some of the pioneering work includes GroupSketch [1] , VideoWhiteboard [2] , and Liveboard [3] . In these preliminary systems, users and their drawings are captured by cameras, transmitted and projected to remote screens. An issue of these systems is that only screenshots are transmitted, thus they lack the flexibility to provide interaction among people geographically dispersed. Subsequently, researchers developed more elaborate native desktop collaborative applications. For instance, Booth et al. [8] proposed a "mighty mouse" multi-screen collaboration tool, which provides a smooth mouse movement cross-platform, via VNC protocol. A crucial issue of this kind of systems is that they usually require complicated setups on different platforms because they need to have distinct implementations on these platforms. Recently, with the significant enhancement of modern web browsers, especially with the advent of technologies brought from HTML5 standard, such as WebSocket, researchers began to consider web browsers as the backbone for RTC applications. Bentley et al. [9] , [10] gave pioneering work in this direction. They produced a basic shared workspace system across web browsers. Mogan et al. [11] performed a comparison study on three group-aware tools: desktop-based Java tool XCHIPS [12] , browser-based Adobe Flash application ThinkTank, and browser-based AJAX system PowerMeeting. They claimed that the last one has advantages in features, user interface, usefulness, etc. over the others. Schmid et al. [13] employed an XML format to develop a web-based interactive collaborative environment. Now, Google Docs and Etherpad are both rather mature products which enable realtime document editing collaboration among multiple users. Fetter et al. [14] created a collaborative web browsing tool distilling the concept of lightweight interference, transitions, and adoptions. Binda et al. [15] developed a photo-sharing system to stay aware of family members' health. Lee et al. [16] developed a UI design tool to explore RTC in crowd-powered systems. In our study of four prior collaboration systems, they only support very limited types of media, as demonstrated in Table I . Here, Google Docs supports document editing, Google Hangouts supports video watching, SpreadVector [14] is for collaborative web browsing, and Collaboard [17] has a whiteboard functionality. The system described in [7] works for more types of media, even though it still only supports whiteboard and image editing. By contrast, CWcollab supports all types of multimedia listed in the table, especially for presentation purposes. This notable feature differentiates our work from prior work. A crucial feature distinguishing our design from other collaboration systems is the general-purpose distributed collaboration with a uniform interface. As events are transmitted in a standardized message format, this design significantly relieves the difficulty of supporting a new type of media. Figure 1 shows the structure of collaboration based on a uniform interface. Sender flow. For the client who initiates a media event for collaboration, there is a media event capturer module to monitor and capture the event. A media state recorder will be invoked when necessary to record the current media state for possible resynchronization. For example, the state of a video may include the play/pause state, volume, progress, playback rate, related annotations, etc. At the same time, the event is sent to a message serializer, which encapsulates the event into a standardized message and sends it to the backend service. Receiver flow. After the message is routed to one target client via the backend service, a message deserializer takes charge of unwrapping the message and sending information to the media event replayer to update the current media state. Simultaneously, the change of media state may also be recorded in the media state recorder module. With the uniform interface, if a developer would like to add collaboration mechanisms for a new type of media, he/she just needs to follow the structure by adding functions in the media event capturer, media state recorder, and media event replayer. Message serializer and message deserializer should be left intact if the format of transmitted messages is not changed. The feature of extendability significantly relieves the burden of implementing new functionalities for collaboration. The main functionality of the media event capturer component is to capture an abstract media event via DOM events. In our design, by using event propagation, especially event bubbling, which is now supported in all of the modern browsers, we create another layer on top of original media objects to capture these events. We leverage an object-prioritized hybrid approach to capture media events. Specifically, it consists of: • Object-based. Events are captured as DOM events on the media object or a UI component. This provides more flexibility than the traditional position-based approach, as no matter where a media event fires, we only trace the object effected, isolated from the influence of positions. This type of capture scenario is much more often seen than others. Figure 2 illustrates how to capture objectbased video events. In the figure, every control is triggered through a UI component. The media event recorder is used to track the current state of the media. The state of a media block depends on the type of media. A state is a set of key-value pairs. For example, a state of a video contains the video source, current timestamp, muted or not, volume, playback rate, annotations, etc. The key-value pairs can be compressed into messages for further transmission. The main functionality of a media event replayer is to replay events attached with complete media event information. To achieve that efficiently, we design a hierarchical tree structure-Handler Tree-to handle messages, as shown in Figure 3 . An event is propagated in the handler tree via parent-child chains until arriving at a leaf node. During the propagation, the received message is parsed gradually, and finally, in a leaf handler, the message is totally consumed to update the UI if necessary. An example of handling a video play/pause event in CWcollab is demonstrated in Figure 4 . When an event is received by an audience, it is first sent to the root node, also acting as a dispatcher in the handler tree. The next level in the handler tree contains various handlers for corresponding types of media. The message is sent to the appropriate media handler at this level. Then, a specific action handler is invoked to fire the target DOM event. For the example in Figure 4 , the message is first handled by the root handler and dispatched to the video handler according to its media type. Because the event type is button click, the message is next sent to the button click handler for further process. Then the handler reads the target id and triggers the click event on the element with this identifier. The handler also loads the value of current time in the data field to update the current progress of the video to that timestamp. Considering that our system is real-time, the tree is flat to reduce the overhead. With the help of the media event replayer, the events occurring in a collaboration session can be replayed sequentially according to their timestamps. The events occurring in a session are pushed into a queue structure. These events are scheduled to be popped from the queue and sent to the handler tree for replay. In CWcollab, collaboration events are captured and represented by a standardized format of strings, and the strings are transferred among collaboration subjects. There are two different types of messages: media events and control events. For a media event message, it represents an action related to a specific media block, such as scrolling a web page. Table II demonstrates the possible fields in a media event message. By contrast, a control event message represents an action involving controls in a collaboration session, such as resynchronizing the media state. Table III shows the fields for a control event message. The responsibilities of a message serializer and message deserializer are to wrap a media message into a standardized format of string and to read the message from such a string, respectively. To maintain consistency, the two components share the same message format. We created a web application 1 based on CWcollab to provide a collaborative presentation service, including account management, materials preparation, presentation synchronization, and replay. The backend service is implemented in Node.js, which is an event-driven, non-blocking, and crossplatform JavaScript run-time environment. We use MongoDB, a document-oriented NoSQL database, to store presentation data. We also use Redis, which is an in-memory key-value database, as a message broker. The participants of a collaboration session can download media materials beforehand or the presenter can dynamically add materials in a session. In the application, manipulations on the material in the presentation panel are synchronized to other users' panels. Currently, screen sharing products such as Zoom and Google Hangouts are widely used to provide basic collaboration support in video conferencing. In this section, we compare Google Hangouts with CWcollab from multiple perspectives. Although we didn't evaluate other video conferencing products in details, we reckon that they share similar mechanisms for screen sharing. We performed a series of experiments on multimedia, including videos, PDF documents, images, and web pages, to measure the network bandwidth usages in one minute of Google Hangouts and CWcollab, on the presenter side. Instead of transferring video chunks during a presentation, the static materials can be loaded beforehand, and only minimal events information is transferred in a session. We found that CWcollab requires a much lower bandwidth usage, as illustrated in Figure 5 , 6, 7, and 8. Note that the y-axis units are different in these figures. Figure 5 shows the network bandwidth consumption of both Google Hangouts and CWcollab for presenting a video. Figure 5a shows the Google Hangouts network bandwidth consumption is around 60KB/s -100KB/s, which is mainly for transferring video frames. However, CWcollab presets the static media-the video-in this experiment, and the network bandwidth consumption is only around 0B/s -1KB/s in Figure 5b . The bandwidth consumption comparison is summarized in Figure 5c . Similarly, Figure 6 shows Google Hangouts utilizes around 60KB/s bandwidth while CWcollab only needs a maximum 3KB/s for presenting a PDF document; Figure 7 shows Google Hangouts utilizes around 70 KB/s and CWcollab only needs a maximum 2KB/s for presenting an image; Figure 8 shows Google Hangouts needs around 100KB/s and CWcollab needs almost 0B/s most of the time (the two peaks are for loading web pages). To summarize Figure 5 -8, we can see that in the measurement of one minute, most events only require hundreds of bytes per second. The most expensive action is the free drawing event whose bandwidth usages can be as high as around 3KB/s. However, the usages are still much lower than those of Google Hangouts. We also summed up the bytes transmitted in this one minute and showed the results in Table IV . CWcollab transmits very few bytes in a presentation, especially noticing that in the collaboration session except web pages, in one minute, the total bytes transmitted in CWcollab is less than 10 KB, because CWcollab leverages an object-prioritized approach to capture media events and represent them in simple messages. As the web pages have to be loaded dynamically, even though loading pages costs a large amount of network bandwidth, CWcollab still requires much lower bandwidth than Google Hangouts. Besides bandwidth usages, there are some other differences between screen sharing tools and CWcollab. Because the mechanism of screen sharing tools is to capture and broadcast the display, it cares nothing about what application the user is synchronizing. As a result, it is straightforward for these tools to support multimedia in web browsers and desktop environment. By contrast, our platform is web-based. To support another type of media, we need to instrument necessary control to achieve the collaboration. Additionally, CWcollab can be a supplement to current screen sharing products, considering that it provides an efficient platform to support message-based collaboration on multimedia. In this paper, we proposed a context-aware web-based collaborative multimedia system-CWcollab. It supports collaboration on multimedia and uses simple messages to represent media controls. We demonstrated the design methodologies of a distributed collaboration framework and discussed the implementation of architectural components. We compared CWcollab with screen sharing tools such as Google Hangouts. Our evaluation results showed that our tool has a lower bandwidth usage than that of Google Hangouts. We posit that this work could serve as a uniform platform for efficient general-purpose multimedia collaboration. Group sketch: a multi-user sketchpad for geographically-distributed small groups Videowhiteboard: video shadows to support remote collaboration Liveboard: a large interactive display supporting group meetings, presentations, and remote collaboration Real-time collaborative coding in a web ide Richreview++: Deployment of a collaborative multi-modal annotation system for instructor feedback and peer discussion Full-body webrtc video conferencing in a web-based real-time collaboration system An interactive pervasive whiteboard based on mvc architecture for ubiquitous collaboration The mighty mouse multi-screen collaboration tool Designing a system for cooperative work on the world-wide web: Experiences with the bscw system The world wide web as enabling technology for cscw: The case of bscw The impact of web 2.0 developments on realtime groupware A cooperative hypermedia approach to flexible process support for managing distributed projects Real-time collaboration through web applications: an introduction to the toolkit for web-based interactive collaborative environments (twice) Lightweight support for collaborative web browsing through spreadvector Phamilyhealth: A photo sharing system for intergenerational family collaboration on health Exploring real-time collaboration in crowd-powered systems through a ui design tool Collaboard: a novel interactive electronic whiteboard for remote collaboration with people on content