I’ve recently took on a contract position with a London-based startup looking to develop a video discovery platform being a more refined version of YouTube, featuring only curated content.

Aiming to attract users from other content discovery platforms, the mobile application (built using Ionic Framework) was designed to be super immersive and interactive, featuring quite a few animations and sound effects.

We’ve found the development of UI features particularly fun as well as challenging. On few occasions, we came up against a brick wall, encountering Webview related limitations that had to be workaround implementing certain features in native Java or Objective-C code. That is not to say that Webview based frameworks are generally inferior to native development. Building mobile applications for the past couple of years in several different frameworks, I’d be inclined to say that Ionic Framework is an excellent choice, for majority of the projects (considering e.g. ease of development and future maintenance) . However in some less common cases, certain requirements can be difficult to implement without the aid of native development.

One example of this was development the full screen video background. A similar effect had been used by Spotify and Uber apps, but our came with an extra twist: it would smoothly move to the left and right as user scrolled or swiped through the pages. The clip below shows what I exactly mean by that (since a clause in my contract prevents me from sharing the actual designs, it’s just a simple prototype without the actual content and styling):

1. HTML5 video element

I’ve started by implementing the video in HTML5, which was fairly simple. For brevity, in this blogpost I’ll mainly look at the actual method of playing the video and skip the details of the scrolling implementation.

There were few things take into account when preparing the actual video file:

  • Small file size, to minimize the application load time and total binary size

  • Video format. While the mp4 format (H.264) is widely supported by desktop browsers, its support on mobiles is patchy. We’ve found that the same video would work perfectly fine on a device running Android 4.2, but would fail to play on Android 4.0.4. It’s not sufficient to simply encode the file in H.264 - to make sure the video works across different devices, it’s important to encode it with specific settings (more details can be found here).

  • In order for the video to cover the entire screen, its resolution had to be wide enough to support aspect ratios of all devices (iPad’s 4:3 was an edge case).

Aside from that, it was as easy as dropping the <video> element in the template and adding a style removing the element from the document flow and placing it in the background:

Page template

<ion-view>
    <video autoplay="autoplay" id="background-video"></video>
</ion-view>

Template styles

#background-video {
    position: fixed;
    right: 0;
    bottom: 0;
    min-width: 100%;
    min-height: 100%;
}

2. Canvas based approach

While this naïve method worked fine in the browser and on Android, I discovered that it didn’t work as expected on iOS where the video opened in a full screen mode, covering the application elements. This behaviour usually can be controlled using the ‘webkit-playsinline’ video attribute that allows to play the video inline as opposed to playing it full screen. However due to a Webkit bug (1, 2), support for this attribute was broken in mobile Safari.

To workaround this issue, instead of using a <video> element, I decided to render the video using HTML5 canvas. This was done by fetching consecutive frames from the video element and manually drawing them, one by one, on the canvas. The video was still being loaded into the <video> element, but only for the purpose of accessing its frame data. I’ve found couple of plugins built for this specific purpose (1, 2), however in our case it made more sense to implement a basic canvas rendering on our end, in order to integrate the video scrolling feature with it.

Page template

<ion-view>
    <video autoplay="autoplay" id=”video-background”></video>
    <canvas id="canvas-background"></canvas>
</ion-view>

JavaScript snippet responsible for video drawing

var FRAMES_PER_SECOND = 60;
var lastDrawTime;
var videoElement = $('#video-background');
var canvasContext = $('#canvas-background').getContext('2d');

function loop() {
    var time = Date.now();
    var elapsed = (time - lastDrawTime) / 1000;

    var timeToDrawFrame = elapsed >= (1 / FRAMES_PER_SECOND);

    if (timeToDrawFrame) {
        videoElement.currentTime = videoElement.currentTime + elapsed;
        canvasContext.drawImage(videoElement, 0, 0, videoElement.videoWidth, videoElement.videoHeight);
        lastDrawTime = time;
    }

    var endReached = videoElement.currentTime >= videoElement.duration;

    if (endReached) {
        videoElement.currentTime = 0;
    }

    requestAnimationFrame(loop);
}

loop();

3. Native implementation

The video seemed to have performed well, but under a closer look we discovered that on some older devices (eg. Samsung S4, iPhone 5s), the video frame rate was noticeably lower (at times, below 30 FPS as shown in the profiler screenshot below) particularly when the video was being scrolled.

FPS issues

I’ve tried couple techniques of improving the video performance (analysing function execution and rendering time with CPU and Timeline profilers, decreasing the video size and resolution, temporarily turning off any Angular bindings to make sure there isn’t anything external affecting the performance). After a few trials giving faint results, I decided to try a completely different approach and build a quick prototype with the video implemented on the native side.

I started off with iOS and made few modifications to the Objective-C code. I’ve extended Cordova’s CDVViewController and played the video using AVPlayerLayer, placed behind the WebView.

CDVViewController.h

@property(nonatomic, strong) AVPlayerLayer *videoLayer;

CDVViewController.m

- (void)viewDidLoad {
  [super viewDidLoad];

  [[NSNotificationCenter defaultCenter]
      addObserver:self
         selector:@selector(replayVideo:)
             name:AVPlayerItemDidPlayToEndTimeNotification
           object:nil];

  [contentLayer addSublayer:self.videoLayer];
  [self.view.layer insertSublayer:contentLayer atIndex:0];
}

- (AVPlayerLayer *)videoLayer {
  if (!_videoLayer) {
      
    NSString *movieRelativePath = @"img/background-video.mp4";
    NSString *movieAbsolutePath = [(CDVCommandDelegateImpl *)_commandDelegate
        pathForResource:movieRelativePath];
    NSURL *movieURL = [NSURL fileURLWithPath:movieAbsolutePath];

    _videoLayer = [AVPlayerLayer
        videoLayerWithPlayer:[[AVPlayer alloc] initWithURL:movieURL]];
    _videoLayer.frame = CGRectMake(0, 0, self.view.frame.size.width,
                                   self.view.frame.size.height);

    [[NSNotificationCenter defaultCenter]
        addObserver:self
           selector:@selector(replayMovie:)
               name:AVPlayerItemDidPlayToEndTimeNotification
             object:[_videoLayer.player currentItem]];

    [_videoLayer.player play];
  }
  return _videoLayer
}

- (void)replayVideo:(NSNotification *)notification {
  [self.videoLayer.player play];
}

Also, since Ionic by default uses a dark background, in order to make the AVPlayerLayer visible, the background had to be made transparent:

Template styles

.platform-ios .pane {
    background-color: transparent;
}

That did the trick! The video played using AVPlayer worked very smoothly, even on older devices and it didn’t affect the visible performance of the rest of the application. When I have some free time, I’ll look into packing up the above code into a simple reusable Cordova plugin including support for Android.

While I haven’t covered the implementation details of the scrolling, it was an quite an interesting feature to tackle as well. In short, since the video was supposed to scroll along with the Ionic pages (or slides) - with the same transition effects (e.g. tweening), the video position had to be bound to the page position. For example, if the page slided 100px to the right, the video would move 100px to the right as well.

The page position could be retrieved by subscribing to on-scroll event on <ion-content> elements (it was a bit more tricky for <ion-slide> element since it doesn’t expose such event). To actually move the video, the page position had to be passed from JavaScript to the Objective-C, where the actual change to of video position took place. Communication between web and native layers can be achieved by writing a custom Cordova plugin (you can find more details in Apache docs).

Page template

<ion-content on-scroll="pageScrolled()"></ion-content>

Controller file

$scope.pageScrolled = function () {
    var scrollOffset = $ionicScrollDelegate.getScrollPosition();
    cordova.exec(
        null,
        null,
        "BackgroundVideo", // Name of the native class
        "setPosition",  // Name of the native method
        [scrollOffset.x, scrollOffset.y] // Parameters passed to the native method
    )
}

Objective-C plugin file

- (void)setPosition:(CDVInvokedUrlCommand *)command {
  NSArray *arguments = command.arguments;
  NSNumber *offsetX = [arguments objectAtIndex:0];
  NSNumber *offsetY = [arguments objectAtIndex:1];
  playerLayer.position =
      CGPointMake([offsetX doubleValue], [offsetY doubleValue]);
}