https://public.etherpad-mozilla.org/p/wr-plan
Proposed plan for using WebRender as a Layers backend:
https://public.etherpad-mozilla.org/p/Gecko-WebRenderer
- The goal is to write code that we're going to use in the final state, and still have something workable sooner
- We had an earlier idea to have WebRender compositor, which would mean having a WebRender impleGDGFGFDmentation of the Compositor AP
- This plan instead wants to replace more of the layers system by WebRender:
- FrameLayerBuilder stays and creates layers using the LayerManager API
- We create a new LayerManager implementation called WebRenderLayerManager, which replaces ClientLayerManager and would be used by FrameLayerBuilder
- WebRenderLayerManager knows how to serialize the laASDyer tree to a WebRender display list, e.g. turning ASDPaintedLayers into WebRender image display items
- We then send the WebRender display list to the compositor thread and render it using WebRender
- We'll need to write these things that we'll need for full WebRender in the end anyway:
- WebRender APZ
- texture sharing
- WebRender IPC
- OMTA
- OMTV test abc tesbac
- These things would stay:omtaASDASD
- PaintedLayers, FrameLayerBuilder and Invalidation
- (PaintedLayers are just texture-shared images that get rendered by WebRender image drawing)
- Tiling
- Buffer rotation
- Transition
- We can add more and more supported web render display item types, by creating layer types for them
- For example, for text, we'd add a TextLayer, which then gets turned into a WebRender text item on the client ly reduce the distance between the APIs, until in the end state we can have one layer type per webrender display item type
- (this means that, since we'll sometimes have multiple wr display items per nsDisplayItem, we'll need a way to make an nsDisplayItem create multiple layers)
- At that point, we remove the intermediary and directly create WebRender display items from nsDisplayItems
- We'll also need invalidation on the WebRender side at that point, and then we remove FrameLayerBuilder
- Advantages:
- Less new code that we're not going to use
- Makes the path after this point more parallelizable, because it pulls most of the bottlenecks to the front
- Can get tests passing early and keep them passing
- Will prove all the connections to e.g. the widget system, event handling stuff
- Changes the type of bugs we're going to get, for example if something paints wrong, the bug should be known to be where it is in the pipeline (based on what we've changed)
- If some part will take longer, e.g. video, then it wouldn't hold up this incremental step, we'd just make it slower (?)
- Disadvantages:
- Being incremental makes it easier not to finish (we might end up kicking the old infrastructure around forever)
- FAQ:
- Will we be able to ship this intermediate step?
- Most likely not. This is mostly intended so that we can work on more pieces sooner, and keep tests passing during the process.
- Will it speed things up?
- Not sure, but probably not at first. In the beginning it will make many things slower, e.g. we won't have APZ until we have APZ for webrender.
- Is there anything smaller that is better and shippable?
- What code will be writing for this intermediate step that we don't need in the end?
- WebRenderLayerManager
- additional layer types for any new webrender display types that we add support for (those will only be implemented for WebRenderLayerManager and just be small value containers)
- What directories in gfx/layers/ would stay around?
- apz, basic
- new webrender directory
- maybe layerviewer + protobuf with lots of changes
- Isn't layers currently designed with the assumption that there won't be "too many" of them? Will there be perf regressions if we start increasing the numbers/types of layers?
- This is irrelevant, because the WebRenderLayers are going to be converted to WebRenderDisplayItems almost immediately
- At least if we're going with option 2. If we go with option 3, i.e. converting ACM'd layers on the compositor side, then there will be perf problems with lots of layers.
- How will this coexist with existing layer manager implementations? Will the new layer types have to be supported in existing layer managers? (We should try to avoid littering FrameLayerBuilder with layer-manager-specific conditions)
- The other layer managers can just return null from the CreateXXXLayer() calls, and FrameLayerBuilder would fall back to the path that does not create layers of these types.
- Or the display items could, in GetLayerState(), check the layer manager type and return LAYER_NONE instead of LAYER_(IN)ACTIVE
- Which process will webrender live in, and what's the data that gets sent over?
- The current plan is to put webrender into the GPU process, and send over the webrender display list.
- One alternative would be to run webrender in the content process and send over GL commands. But then we'd still need some way of compositing the result from two different processes.
- What about texture sharing?
- Not sure, but we'll probably have to add it at some point, because we'll still have things like SVG ( / canvas? ) for some time that need to be rasterized on the content side.
- What does the transition plan look like?
- We do something a little bit like this for ImageLayers today
- We have a CanOptimizeToImageLayer() that we call it determines if we can create an image layer
- We can create something like this for WebRender display items
- We'll be able to track how much fallback we have happening and reduce it incrementally
Serialization bottlenecks
- texture sharing somewhat blocks video
- apz
- widget
https://public.etherpad-mozilla.org/p/wr-plan
Workitems for initial prototype of WRLayerManger
-MaskLayers we need mask image support in WebRender
-bring up c bindings for WebRender
-Widget stuff
-Probably best to start by bringing things up in-process with webrender running synchronously. i.e. make something akin to basic layers
Questions for webrender people:
- Why is a scroll frame different from a regular frame? What happens if we have both a regular paint / display list update and a scroll position change? Do the two get coalesced into one composite?
- First let me apologise for some of the naming conventions - they make sense to me coming from a game background but are often ambiguous with web terminology :)
- If I understanding what you're asking, a normal frame is when content of the display list has actually changed and we need to rebuild GPU data structures etc.
- In a scroll frame, there's no involvement from layout etc, all the data is exactly the same, just the visible viewports are changing.
- There is definitely some coalescing that goes on - for instance, if we very quickly receive 3 new regular frames, we coalesce then. We can tweak this behaviour with scroll frames too if it doesn't work quite right now.
- What is the pipeline ID for? How much does it matter?
- In servo, a pipeline ID identifies a top level page or an iframe. For instance, if you have a page that contains 2 iframes, there would be 3 pipeline IDs in the stacking context tree to be drawn (one for the root page, one for each iframe). I'm not sure how this fits in to gecko terminology.
- What thread is the notifier called on? How should it react to the call?
- It's called on the webrender backend thread. In Servo, we use this to call a (thread safe) function that wakes up our main thread (which may be blocked on waiting for UI events). When that thread gets woken up it knows to tell WR to draw the next frame.
- Can you delete an image before you submit a new display list without that image?
- Yes - the idea is that WR handles images (and fonts) by key/id for this exact reason. The behaviour should be that the display item referencing a non-existent resource gets dropped and not drawn. If it doesn't work that way currently, it's a bug.
- Why does a stacking context have multiple display list IDs?
- This is due to the way servo (used to) build display lists in parallel - where there is one "display list" per stacking context (I think). Probably best to get pcwalton to comment on this as I'm a little fuzzy on the details.
- Is data being retained between frames based on the display list IDs or the stacking context IDs, or can we regenerate these IDs on every frame?
- That data is retained when scrolling (which I believe is similar to APZ) - there is no no data arriving from the layout thread(s) in that case. When you provide a new layout (i.e. via SetRootStackingContext), any existing data is thrown away and so it's fine to re-use IDs.
Questions for us:
- The WR display list is processed on a different thread. How do we want that asynchronicity tie in with the rest of Gecko / Gecko's painting?
- (Matt) Is there any advantage to making WebRenderLayerManager work as a replacement for LayerManagerComposite (after layers have been sent to the compositor) rather than ClientLayerManager initially?
- It seems like we could write it this way and then transition it to the client in the future with very little extra work (it's just a LM impl, doesn't matter if the user is FrameLayerBuilder or LayerTransactionParent).
- This would give us working texture sharing (TextureClient is *huge*, and most of the work we do to serialize all the different surface types would still probably need to exist), async video, APZ etc
- omta
- If this is workable, then we could get some initial data (and maybe even real wins) while we work on WebRender IPC, WebRender APZ etc etc
- I don't have any real objection to this plan. I am concerned about the additional serialization overhead though.
------------------
Ideas for software webrender backend:
Invalidation
1. Screenspace tile based invalidation (i.e. has the tile changed)
2. Scroll layer invalidation. (interleaving things are invalidated everytime)
3.
Invalidation roots
- these would be scrollable things, transformed things. These would retain painted versions, but only what's visible on the screen
- invalidation roots would be largely static
- stuff on top not part of the invalidation root is repainted every composite. i.e. if something is inbetween, we propose repainting everything in that area on every composite
- e.g. imagine you have some scrollable content that has something inbetween that does not scroll:
- when the scrollable thing scrolls, we repaint the area
changing content on a page can be handled by doing something like DASDLBI with per tile. We can do this per invalidation root as we needed.
Don't worry about position fixed content? Can we just redraw in time?
How to convert a display item type to webrender:
- Find the type that you want to convert in nsDisplayList.h or in nsDispAlayItemTypesList.h.
- Find the nsDisplayItem subclass of that type.
- Add an implementation of nsDisplayItemYourType::GetLayerState and make it return LAYER_ACTIVE for webrender layer managers and LAYER_NONE for the other layer managers.
- Go to Layers.h and add a subclass of Layer for the type that you want to add.
- Go to WebRenderLayerManager.h and add a CreateYourTypeLayer() method (all the other layer managers should return null).
- Add nsDisplayItemYourType::BuildLayer implementation that calls aManager->CreateYourTypeLayer() and sets the correct values on the layer.
- Before step 6: Figure out what values the layer needs to know about, by reading all the code that is called from nsDisplayItemYourType::Paint.
- In WebRenderYourTypeLayer::RenderLayer, add conversion code to build WebRender items of the right type with the right parameters. Might need new bindings in bindings.rs / webrender.h, or even new webrender capabilities.
Matt has already created layer types for Text and Border, for the simple cases.
Converting these things is going to be hard:
- Text with SVG glyphs
- text decoration
- text shadow
- text selection
SVG
- filters
- - build descriptions ahead of time
missing things
- multiple mask images blended together
- clipping to multiple paths
svg patterns
- % patterns make things difficult because we'd need to do layoutish things
- we could flatten out display lists but we might get to many display lists
Idea for supporting text:
The problem is that text rendering can do lots of things that webrender won't support at first. Ideally, you'd have a function CheckTextLayerFeasibility that either returns an array of glyphs or false. But the logic for creating such a check would be really complicated; you'd end up duplicating all of the control flow that is in the regular rendering path.
One idea would be to call the regular text rendering code with a recording DrawTarget during display list construction, and then analyze whether there are only FillGlyphs calls in the recording. If yes, extract the array of glyphs; if no, hold on to the recording DrawTarget and execute the drawing commands during actual painting.
Another idea would be to create a class that initially just wraps a DrawTarget, and that gets passed into text painting code instead of the actual DrawTarget. (This would require lots of type renaming but shouldn't change behavior.) This wrapper class could have an internal (lazily-created) DrawTarget and an array of glyphs. If FillGlyphs is called on the wrapper, the parameters would be added to the glyphs array; if any other drawing method is called, the internal DrawTarget is created, existing recorded FillGlyph calls are executed on it, the FillGlyphs array is discarded, and the current + all future drawing commands is executed directly on this DrawTarget.
In the case where WebRender is disabled, the text rendering code would be called at the normal place during painting, and the wrapper class would just forward all calls to the DrawTarget that backs the PaintedLayer that we're drawing to.
Types: (tag with inactive, started, converted, enabled)
- ALT_FEEDBACK: - https://bugzilla.mozilla.org/show_bug.cgi?id=1345388
- for images that are loading / broken, draws a border, or a placeholder image, or a different placeholder image, or the alt text
- detect the state during display list construction, and then create items of the appropriate types.
- BACKGROUND
- can draw a background image or a linear gradient or a radial gradient
- might want to consider splitting those up into separate types
- BACKGROUND_COLOR
- creates a solid color rectangle
- THEMED_BACKGROUND - Keep as painted layer
- for theme drawing; need to create a surface, have the theme draw into it, and give it to webrender as an image.
- BLEND_CONTAINER / BLEND_MODE
- container items which we already know how to create layers for; webrender already supports blend modes on stacking contexts.
- BORDER
- can do simple border drawing, multi-colord border drawing, and border-image drawing
- might want to create a new item type for border image
- if multi-colored borders (used for the -moz-border-top/right/bottom/left-colors properties) can be expressed as multiple regular border items, stacked on top of each other, maybe we should split those up during display list construction
- BOX_SHADOW_OUTER / BOX_SHADOW_INNER: - outset bug 1347727, inset bug 1349843
- extracting the data should be straightforward. requires new layer type and is already supported by webrender (BoxShadowDisplayItem)
- BULLET: - bug 1339683
- Supports disc, circle, square, disclosure triangle and image.
- disc, circle and square are drawn as glyphs -> text item
- disclosure triangle is a triange path that is filled -> path item
- image -> image item
- BUTTON_BORDER_BACKGROUND: - 1350182
- can hopefully be split somehow into border + inner box shadow
- TYPE_BUTTON_FOREGROUND:
- can be two border items, need to pass a style context (or an nsStyleBorder) to the constructor because outer and inner focus border can be different
- CANVAS_FOCUS:
- border, we should change that to just be a border item
- CARET: bug 1342276
- CHECKED_CHECKBOX:
- either fills a path (checkmark) or a rect ("minus", when undetermined) -> path item / colored rect item, depending on state
- CHECKED_RADIOBUTTON:
- ellipse, should use a webrender path item
- CLEAR_BACKGROUND:
- colored rectangle (transparent) with operator source
- COLUMN_RULE: bug 1344079
- COMBOBOX_FOCUS:
- EVENT_RECEIVER, LAYER_EVENT_REGIONS
- depends on what kats decides to do for event regions - todo follow up
- FIELDSET_BORDER_BACKGROUND
- can be split into border and background
- FIXED_POSITION
- just a container, the fixed information could be put on a stacking context
- STICKY_POSITION:
- FRAMESET_BORDER:
- FRAMESET_BLANK:
- HEADER_FOOTER:
- draws text. We could create a display item that just accepts text in its constructor and draws it / creates a TextLayer.
- IMAGE: x
- LIST_FOCUS:
- draws a focus border using nsCSSRendering::PaintFocus, can be a border item
- OPTION_EVENT_GRABBER:
- doesn't paint anything, don't need to do anything
- OUTLINE: - 1348755
- either theme drawing or border, split into two display item types
- OWN_LAYER
- can be flattened away, or can be a stackingcontext with the extra data that is attached, e.g. for scrollbar thumbs
- PLUGIN
- PLUGIN_READBACK
- dead in a webrender world and potentially sooner (only needed needed for windowed plugins, async plugin painting is already used 50%)
- PLUGIN_VIDEO
- RANGE_FOCUS_RING
- REMOTE:
- RESOLUTION:
- is a container, already supported as a containerlayer
- SCROLL_INFO_LAYER:
- SELECTION_OVERLAY
- SOLID_COLOR: - 1351510
- SOLID_COLOR_REGION - 1351511
- list of colored rectangles
- SUBDOCUMENT:
- already a container layer
- MASK
- stacking context with a mask
- FILTER
- stacking context with a filter
- SVG_OUTER_SVG:
- we need to keep this one.
- image: paints the whole <svg></svg> element as one image
- SVG_TEXT
- calls PaintText on an nsTextFrame, so we need to do the same thing here as for nsDisplayText
- The DrawResult stuff seems unnecessary
- TABLE_CELL_BACKGROUND, TABLE_ROW_BACKGROUND, TABLE_ROW_GROUP_BACKGROUND, TABLE_BORDER_BACKGROUND
- will create regular background items when andrew is done
- TABLE_CELL_SELECTION
- is a border (with "shading" - just make it a simple border without the shading)
- maybe remove the feature? bug 1234067
- TEXT: 1329314
- see above for the text plan
- TEXT_OVERFLOW - 1351508
- draws text (ellipsis or other text) with an optional text shadow, same solution as for the other text display items
- TRANSFORM, PERSPECTIVE, WRAP_LIST:
- containers that already support creating ContainerLayers
- VIDEO
- REFLOW_COUNT
- XUL_EVENT_REDIRECTOR:
- nothing gets painted, don't need to worry about this one
- XUL_GROUP_BACKGROUND
- can be refactored to create either one border item or three border items with clips.
- XUL_IMAGE
- XUL_TEXT_BOX
- XUL_TREE_BODY
- we'll need to keep this one, or do lots of work to break it down to the individual item types, see 5.2 in jeff's list below
- XUL_TREE_COL_SPLITTER_TARGET
- just used for hit testing, doesn't paint anything.
- XUL_DEBUG:
- Debug only, calls nsBoxFrame::PaintXULDebugOverlay, which calls DrawSpacer, which draws lines and rectangles
- MATHML_BAR
- MATHML_CHAR_BACKGROUND
- MATHML_CHAR_FOREGROUND
- MATHML_ERROR
- fillRect + the string "invalid-markup"
- MATHML_MENCLOSE_NOTATION
- MATHML_SELECTION_RECT
- MATHML_SLASH
- MATHML_BOUNDING_METRICS
- just does one StrokeRect. This could be a border, or it could be four colored rects
- MATHML_CHAR_DEBUG
- DEBUG_BORDER, DEBUG_IMAGE_MAP, DEBUG_PLACEHOLDER, EVENT_TARGET_BORDER
- debug only, can just fall back to PaintedLayers forever (maybe not forever)
We should figure out all the places where we draw borders and write down what pieces we need. Create a new nsDisplayBorderSomething class that takes an nsStyleBorder in the constructor? Or a rect, border color + border width?
We should minimize the list of display item types by merging some of our display item classes, e.g. all solid color rects (SOLID_COLOR, FRAMESET_BLANK, CARET), all borders (borders, outlines, focus outlines), ...
These classes need to return different perframekeys for the different parts of an nsIFrame that they're used for. For example, if we have a frame with a border and an outline, and we use the same display item type for both, then DLBI needs to be able to differentiate between the two items.
Display items
- Images
- How do they get to the webrender process? Currently it looks like ipc-channel copies the data
- How do we avoid not using more memory than gecko currently does for images?
- i.e. We'd rather not have two copies of every image on a page in memory
- Create some image memory stress tests
- make sure we handle keeping things around for the unaccelerated side
- Current plan is to add and remove images based on Lock/Unlock()
- nsDocument::AddImage
- nsDocument::RemoveImage
- high quality downscaling
- SVG
- https://public.etherpad-mozilla.org/p/svg-perf
- Two possible approaches
- Native support for SVG display items
- Probably good to bring up servo's SVG renderer as soon as possible
- What would the performance of this approach be? (Probably not worse than cairo software rendering)
- For svg geometry frame it may be worth special casing the following
- Treat it like canvas
- Text
- Font memory management may be a concern
- Fonts
- XUL
- Most of XUL should be pretty easy
- XUL Tree (this could be lowered to other display items ahead of time)
- Seperator
- Row
- Column
- DropFeedback
- ProgressMeter
- CheckBox
- Text
- Image
- Cell
- Twisty
- Canvas
- HW accelerated canvas
- texture sharing of some sort (mchang)
- Video
- DXVA
- IOSurfaces
- Android stuff
Misc
- windows ipc channel port
Servo missing functionality
ImageBridge
WebRender missing capabilities
- Subpixel text
- elaborate border rendering
- svg filters
- directwrite
- background-repeat: space (https://github.com/servo/webrender/issues/363 )
- clipping to text (https://github.com/servo/webrender/issues/360 )
- subimage repeat ( https://github.com/servo/servo/issues/13003 )
- hit testing (https://github.com/servo/webrender/issues/411)
- mask images (https://github.com/servo/webrender/issues/405)
- mutliple windows (https://github.com/servo/servo/issues/13338)
- timed async images/pick video frame on composition
- radial gradients
- alpha masks
- anti-aliased edges on transforms
- blend modes
- plane splitting
- Color Management (not required for shipping_
Concerns
- fatness of WR display items for something like SVG (they'll be much fatter than nsDisplayItems which mostly just contain pointers to the frames)
- memory usage overall
Probably worth doing a bit of a WR in gecko prototype:
- and then throwing the thing away and doing it more properly
Overall, nsDisplayItems are very thin and heavily tied to their frames. by the way, I never said that it wan't true. By the time we actually get to drawing there's quite a bit of stuff that happens inbetween at least for image drawing. It may
make sense to make the cut over to webrender at a lower level in the drawing process.
Code that's being replaced:
gfx/layers/*
bindings:
https://github.com/servo/webrender/issues/374
Gecko fixes needed:
- display item per background layer (all that's missing now are tables)
- Remove legacy background layer drawing
- MathML
- XUL Tree
- Masks (how to do this a question mark)
- Border collapse tables
Risks:
- we lose smooth scrolling on pages that are complicated to draw every frame
- (vlad) assuming nothing actually changed, can we cache on the WR side as a performance optimization?
- requires either full FrameLayerBuilder-style layerization, or something simpler as discussed in the software rendering section (retain as one layer and always redraw overlapping parts of the screen)
APZ:
- keep APZCs, replace ACM (AsyncCompositionManager) with a rust + webrender equivalent
- Compositor/APZ hit testing
- Four options, we're going to choose (1) to start. There are some perf tradeoffs between (3) and (4) depending on how expensive WR precise hit testing is, and how often we need to call it relative to the number of transactions
- 1. Dumb c++ HT, main thread fallback <-- this is the one that's closest to what we have now, and probably what we will go with initially
- 2. Smart C++ hit test (needs new c++ code for precise hit testing, operating on the WR display items). <-- this one seems like something we should avoid
- 3. WR hit testing (needs new code for precise hit testing) <-- seems like where we want to end up finally
- 4. Dumb C++ hit test, WR fallback (needs new code for precise hit testing) <-- this would be good second step to migrate towards a fully WR hit test (option 3)
- on a WR transaction the hit testing structure will be built and passed to the C++ code along with the scroll metadata information
- with the "new WR plan" (i.e. using a layers backend) this is simplified because we can reuse the existing code that puts event regions on layers, and just send those event regions to the APZ in the form of WR items
- we will need to add new WR display items to represent areas with touch listeners or touch-action but no visible content (at least; there might be other scenarios)
- we should add support for "empty transactions" in WR, where just the scroll position changes (within the existing displayport) and we just send that over without rebuilding the display list (so that we get paint-skipping in WR)
- whiteboard pics: http://people.mozilla.org/~kgupta/apz-webrender/
Things to be discussed and put in the right place:
- Gecko IPC vs. Rust IPC (events vs. fail when try, implications for security model, for GPU process restart, etc.)
How to do SVG:
- need a path item so that simple geometry with color fills is just that
- also make path items support stroke?
- SVG supports arbitrary fill styles:
- solid color
- linear gradient
- radial gradient
- pattern, i.e. arbitrary SVG / HTML content that's drawn with repeats
- How to do pattern fills? Draw the pattern contents to an image using legacy main thread painting?
Android catch up(Jamie + Sotaro)
- Build on android
- Sotaro is going to check local build, then find out the amount of work.
- Add pause/resume support for Composition with WebRender
- Affected by threading model
- infrastructure tests running
- ES3 vs ES2(exist in another category)
- Support android SurfaceTexture, SharedSurfaceEGL rendering - 1 week sotaro
OMTA(pchang + Sotaro)
- Add support to Webrender(opacity and scaling)
- It is in webrender features needed from other discussions
- Relate Animation Item and stacking context and check OMTA with scrolling
- layout works
- convert nsDisplayTransform and nsDisplayOpacity
- dispatch animation/animationdata to compositor for sampling
- Sampling animation during composition
- Testing
- The followings related to testing of OMTA
- nsDOMWindowUtils::GetOMTAStyle()
- SetTestSampleTime()
- LeaveTestMode()
webrender features needed from other discussions - glenn / kvark / lsalzman
Threading Model - nical / kvark
- separate thread for the WR Renderer
- split TextureHost between the IPDL and the renderer threads
- hooking up APZ, Video, etc through it