updatable and animatable images

[ansis]

Motivation

Images can be used as icons, fill-patterns and line-patterns. Images can currently only be static (with some update support on -native). Being able to animate these images could help make the map more dynamic, emphasize certain features and invite interaction.

Dynamic images could be used for:

• playing a video as a polygon background
• playing a video as an icon
• animating an icon using custom rendering code
• modifying icons in response to user interaction

Design Alternatives

A) push update + beforerender event

Native supports updating images by calling map.addImage again with the same name. We probably want to do the same in -js. Or something similar and add a separate method for updating images:

This could be used for animating icons if we also add a renderstart event to let users push updates before each frame.

map.addImage(name, image);
map.on('renderstart', function() {
    if (animating) {
        map.updateImage(name, newimage);
    }
});

This is the simplest design alternative but leaves more to the user. The user needs to handle adding and removing event listeners when playing/pausing animation.

An alternative name for the even could be renderstart.

B) pull updates with events

We could define a more fleshed out UserImage interface. Users could (optionally) inherit from Evented when defining their image. We could listen to play, pause, seek, reset events and pull in updates when we need to by calling a getImageData method that would render the image.

class MyAnimatedIcon extends mapboxgl.Evented {
    constructor() { this.paused = true; }
    play() { this.fire('play'); this.paused = false;}
    pause() { this.fire('pause'); this.paused = true; }
    getImageData() { /* return rendered image. Called on each frame while icon is playing. */ }
}

const icon = new MyAnimatedIcon(...);
let playing = false;
map.addImage('animated-icon', icon);

...

function onClick() {
    if (icon.paused) {
        icon.play();
    } else {
        icon.pause();
    }
}

C) pull updates with polling

Similar to B), but instead of events have a single boolean animating property that the map polls on every frame. This might be slightly less performant for us but simpler for users to implement.

const image = {
    getImageData: function() { /* render image */ },
    animating: false
}
map.addImage('animated-icon', image);

...

function onClick() {
    image.animating = !image.animating;
}

GL icon rendering design

All the above options could be extended to support rendering icons with gl. This could provide better performance for some types of animation. The implementation would be similar to custom layers.

Since framebuffer binding is relatively expensive we would want to be able to render multiple icons into a single texture.

GL A) user provides texture and coordinates

The user would create an atlas themselves and provide us with a texture as well as the coordinates of the icon. We would then copy the icons into our internal atlas.

const image = {
    width: 10,
    height: 10,
    data: glTexture,
    x: 125,
    y: 300
};

GL B) user provides render method

The user would provide a method for rendering the icon with gl, similar to how custom layers work. We would allocate space for the icon in a texture and use gl.scissor() and gl.viewport() to restrict drawing to that space.

const image = {
    width: 10,
    height: 10,
    render: function(gl) {
    }
};

Design

I think we should implement design 1A). The update image method would be useful one of updates and -native parity. The renderstart event would be simple to add and maintain.

B and C could be useful later but adding them now could be overdesigning. I'd prefer C to B right now because it is simpler.

I think we can initially not support rendering with GL and only consider this if cpu rendering turns out to be a performance issue for our users. GL rendering would come with the same concerns as custom layers on -native (how switching to metal would be handled, etc).

Mock-Up

const image = {
    width: 10,
    height: 10,
    data: renderIcon(userid);
};

map.addImage("asdf", image);

map.on('renderstart', function() {
    image.data = renderIcon(userid, Date.now());
    map.updateImage("asdf", image);
}

example with video:

const video = document.createElement('video');
video.src = '...';
map.addImage('asdf', video);
function updateVideo() {
    map.updateImage('asdf', video);
}
video.on('play', () => map.on('renderstart', updateVideo));
video.on('pause', () => map.off('renderstart', updateVideo));

Concepts

Design 1A does not introduce any new concepts. We'll teach this design with examples that use beforerender to update an image.

Implementation

The renderstart event would fire before the rendering begins for a frame.

map.updateImage would update ImageManager as well as any sprites that are already using that image. It would also need to correctly handle sprites that are being created in workers during that time.

---

@ChrisLoer does design option 1) with no gl rendering sound good for an initial implementation?

[stevage]

Dynamic images could be used for:
playing a video as a polygon background
playing a video as an icon
animating an icon using custom rendering code
modifying icons in response to user interaction

Just wanted to point out that you can achieve the latter three use cases by using Markers instead of Symbols. Then, you get an HTML element that you can do whatever you want with, though obviously with the downside of no collision detection, and it always has to be the top layer.

[agoddard]

Animating an image after adding it, using updateImage, or updating an image within the render method w/ StyleImageInterface both solve part of this, but they seem complex for cases where animated pngs or gifs would otherwise work. AddImage can add an animated gif, however the resulting image is static on the map. Is there an appetite for adding support for apng/animated gifs directly in AddImage?