Part II: Nodejs proxy server -how to speed up running headfull Puppeteer via xvfb

I. Puppeteer

1. Optimalize waitFor functions

Wait for the most specific elements you can.

await page.waitForSelector('button[type="submit"]');
await page.waitForSelector("#mode_search");
await page.waitForSelector("#Until");

2. Intercept and block unnecessary requests

await page.setRequestInterception(true);

page.on('request', (req) => {
   if(req.resourceType() == 'stylesheet' || req.resourceType() ==   
     'font' || req.resourceType() == 'image') {
       req.abort();
   } else {
       req.continue();
   }
});

3. Keep browser running

Launch browser.

async function startBrowser() {

    const browser = await puppeteer.launch({
        executablePath: '/usr/bin/google-chrome-stable',
        headless: false,
        args: ['--no-sandbox']
    });

    browser.disconnect();

    return browser.wsEndpoint();
}

And then for handling requests use connect & disconnect functions.

const browser = await puppeteer.connect({
    browserWSEndpoint
});

// Do something

browser.disconnect();

4. Do not sign in every request

Follows on previous part. Start browser with used userDataDir.

const browser = await puppeteer.launch({
    executablePath: ‘/usr/bin/google-chrome-stable’,
    headless: false,
    args: [‘ — no-sandbox’],
    userDataDir: ‘./userDataDir’
});

Then sign in only once and next requests only check availability of page which requires sign in.

# try go to on page which requires sign in
await page.goto('https://point.vzp.cz/online/online01');

# redirected to sign in page? then does not exists active session and sign in
if(page.url() != "https://point.vzp.cz/online/online01") {

    await page.waitForSelector('button[type="submit"]');
    const submitButtonElements = await page.$$('button[type="submit"]');
    submitButtonElements[1].click();
    await   page.waitForNavigation({ waitUntil: 'load' });
    await page.goto('https://point.vzp.cz/online/online01');

    console.log('signed in');    
} else {
    console.log('already signed in');
}

First sign in can be done when server starts and no request was handled yet.

app.listen(app.get('port'), function () {
    console.log('Server listening on port ' + app.get('port'));

    startBrowser().then(function(browserWSEndpoint) {
        browserWSEndpointGlobal = browserWSEndpoint;

        console.log('browserWSEndpoint', browserWSEndpoint);

        signIn(browserWSEndpoint);
});

5. Package puppeteer vs puppeteer-core

Puppeteer-core is a version of Puppeteer that does not download any browser by default. Puppeteer package by default bundle Chromium. When is used externally installed Google Chrome like below is usage of Puppeteer-core recommended.

const puppeteer = require('puppeteer-core');

const browser = await puppeteer.launch({
    executablePath: '/usr/bin/google-chrome-stable',
    ...
});

II. Xvfb

1. Standard screen resolution

CMD xvfb-run --server-args="-screen 0 640x480x24" npm start

[1] Example project