Automated page rendering revisited - Puppeteer edition

The king is dead, long live the king #

PhantomJS - default headless browser used by many automatic testing solutions - is dead. The only information about that fact can be found in one of 1823 open issues in git repository. Don't blame yourself if you didn't notice. Hopefully you haven't recently started a new project using it or spent few days writing an extensive post detailing workarounds for some interesting edge-case uses.

[EDIT: http://phantomjs.org/ is already updated with this information]

However, it deserves a moment of silence. PhantomJS had a good run, helped solve real development issues and even large brands like Netflix, LinkedIn and Twitter mentioned using it. Who knows how many critical bugs were caught thanks to it?

In the meantime, somewhen last year, Chrome shipped version 59 supporting headless browsing. There is not much to discuss - it's hard to ask for stabler engine than chromium.

Controlling your puppets #

Google ships a framework for controlling Chrome/Chromium under a well chosen name of puppeteer. I have to admit I'm having a blast exploring its API and that's why today we are going to redo some of the functionalities from my previous post.

Puppeteer has also been featured on Google I/O '18, you can watch the session here and see the samples here

Typing your types #

This time we are not bound by PhantomJS engine so we can create a regular Node.js app. I'll be using TypeScript - if you haven't used it before, it's a less-bad JavaScript that gets transpiled to JavaScript but pretty much looks the same like JavaScript. Sarcasm aside, it really helps with whole classes of issues and has a very low starting barrier for anyone proficient in JS and object-oriented paradigm.

I won't be going into details of TypeScript here - samples below barely use any significant features and you should be able to read it just like JavaScript.

Simple rendering #

import * as puppeteer from "puppeteer";

async function run(): Promise {
   const browser = await puppeteer.launch();
   const page = await browser.newPage();
   await page.goto('https://mostlybugless.com');

   await page.screenshot({ path: 'mb.png' });

   await browser.close();
};

run();

Easy!

Viewport and emulation #

Our end-goal is to verify if a website looks good on most common devices. We can easily set viewport of a page with

page.setViewport(viewport)

Besides width and height, it supports additional settings which I'm going to ignore

But one thing to keep in mind is that puppeteer provides

page.emulate

which you can use as a shorthand to emulate any of the already defined devices. Fun!

Single line sanity check:

await page.setViewport({ width: 1366, height: 768 })

Scrolling #

In the previous post I committed some serious crimes against code to create gifs of page scrolling. With full node support this time we could easily integrate one of the more reasonable gif-rendering packages and call it a day, but one issue bothered me - you can't stop a gif. That's nice that you can notice an issue in the middle of a page, but if it's gone in next 200 ms, that's not a great user experience.

Now, puppeteer doesn't (yet) have a direct api to scroll, which is a shame. Still, since we can run anything directly on the page itself..

const stepPoints = await page.evaluate('document.body.scrollHeight')
       .then(scrollHeight => {
           console.log(scrollHeight)
           const adjustedHeight = scrollHeight - step;
           const lastScrollHeigthToRender = adjustedHeight > 0 ? adjustedHeight : 1
           const stepCount = Math.ceil(lastScrollHeigthToRender / step);
           return _.range(stepCount).map(stepNo => stepNo * step);
       });


   for (let point of stepPoints) {
       const path = `mb-${point}.png`;

       await page.evaluate(`window.scrollTo(0, ${point})`);
       await page.screenshot({ path });
   }

With this adjustment, we got

Summary #

Puppeteer is a powerful framework and it's only starting to get attention. I expect to see many interesting use cases in coming months. Also, if you haven't yet tried TypeScript - give it a chance. It will be worth it.

This time I'm not going to completely automate the rendering as in the previous post, but maybe I'll come back to this topic. In the meantime you can get a good start with above code. Enjoy your puppets!