How we added integration tests to our vscode extension using wdio-vscode-service

The why

Not many people use the Kameleoon VS Code extension. With fewer than 200 downloads in the VS Code extension marketplace, it can be challenging to identify and address issues. I’ve had moments when I unknowingly introduced a problem that remained undetected for months. I wish the extension had auto-tests when I first started working on it. As the sole developer responsible for it and being new to the company, the releases were not always perfect, so to speak, and sometimes required hotfixes.

Every time there is a new feature or a fix, I need to explain the nature of the changes to the QA in detail which takes plenty of time, and the fact that it’s tested by them does not free up my own time I would’ve spent testing otherwise, since I still test it all myself too. The only benefit to passing the changes through QA is that I get a look into how it runs in a different environment or a different OS.

Integration tests significantly reduce the time required for regression testing, to make sure that nothing breaks, they are also conveniently run on every major OS: Linux, macOS, and Windows.

The how

First, we need to install the wdio-vscode-service: Follow the official installation guide.

npm create wdio ./

Make sure to select ‘vscode’ whenever you have the option.

It will generate tsconfig.json for you, but if you already have it you can add the types from the documentation, i suggest adding @wdio/globals/types to the types array if it’s not already there, it’ll provide types for the global property “browser”.

{
    "compilerOptions": {
        "types": [
            "Node",
            "@wdio/globals/types"
            "webdriverio/async",
            "@wdio/mocha-framework",
            "expect-webdriverio",
            "wdio-vscode-service"
        ],
    }
}

By default wdio will run every file corresponding to this spec in wdio.config.

specs: [
    './test/specs/**/*.ts'
],

However, for our extension, I chose to manually list the test files.

const testFiles = [
    './test/specs/deployCommands.ts',
    './test/specs/fetchCommands.ts',
    ...
]

const config: Options.Testrunner = {
    specs: testFiles,
    ...
}

So I could exclude them on certain OS where i couldn’t get them to run.

if (!isWindows) testFiles.push('./test/specs/commandExistance.ts')

Finally, in package.json, let’s add the test command.

"scripts": {
  ...
  "test": "wdio run ./wdio.conf.ts",
}

By default wdio will run a separate instance of VS Code for every file you have listed in specs, or for every file that matches the pattern you provided. You can limit the number of concurrent instances with the maxInstances field in wdio.conf.js.

The functionality of the extension is concentrated around calling commands via the command palette (when you press ctrl/cmd+shift+p), and via context menu (when you right click on a file in the explorer).

Let me explain how one of the commands functions.

Test for Deploy experiment command

An experiment consists of files with code (JS+CSS or TS+SCSS) and configuration files, what the deployment command does is taking the contents of the code files, compile them using Gulp and deploy them along with the contents of the config files by calling our public API with a post request.

So the deploy experiment command has 4 steps:

1. When you open a workspace the extension tries to detect a credentials.json file and use it to fetch access token for our public API
2. We click the experiment folder and select deploy experiment in the context menu
3. The extension does its thing, calling Gulp to compile the code files
4. The extension calls our public endpoint using the token and the data from compilation

These are the steps required to use this command with the extension. However, testing it involves a few additional steps.

1. Creating a workspace structure
2. Modify experiment
3. Open the workspace
4. Wait for the token to be fetched
5. Install node_modules
6. Run the deploy command
7. Confirm command success
8. Fetch experiment
9. Assess the fetched experiment

Now let’s look at the code implementation of the test.

describe('Deploy commands', () => {
    before(createFixture)
    before(modifyEntities)
    before(createCredentialsFile(FIXTURE_DEPLOY_DIR))
    before(openFixtureFolder(FIXTURE_DEPLOY_DIR))
    after(deleteFixture)

    it('Should correctly deploy experiment', async () => {
        const workbench = await browser.getWorkbench()

        await hasDisplayedNotification(workbench, `Data for ${FIXTURE_DEPLOY} has been downloaded`)

        execSync('npm i', { cwd: FIXTURE_DEPLOY_DIR })

        await runDeployExperiment(workbench, EXPERIMENT_PATH)
            await hasDisplayedNotification(workbench, 'Experiment successfully deployed')

        removeModifiedEntities()

        await runFetchExperiment(workbench, EXPERIMENT_PATH)
            await hasDisplayedNotification(workbench, 'Data downloaded!')

        const experiment = {
            variationJs: experimentVariationJs,
            variationCss: experimentVariationCss
        }

        assessExperiment(FIXTURE_DEPLOY_DIR, experiment)
    })
})

Here’s the definition of a handy helper function that allows you to check for notifications being displayed.

function hasDisplayedNotification(workbench: Workbench, desiredMessage: string) {
    return browser.waitUntil(
        async () => {
            const notifs = await workbench.getNotifications()
            const messages = await Promise.all(notifs.map((n) => n.getMessage()))

            return messages.some((message) => message.includes(desiredMessage))
           },
           {
            interval: 150,
            timeout: 60000,
            timeoutMsg: 'Notification "' + desiredMessage + ' took too long to display'
        }
    )
}

It checks for the desiredMessage every 150 milliseconds, and if it hasn’t detected it within 60 seconds it will throw timeoutMsg error.

1) Creating a workspace structure

Our extension can’t be used until a workspace is open, and it must have a particular file structure.

That’s why I have created fixtures from which to copy the file structure. Here’s an example of what a workspace would look like for a regular user of the extension.

We need credentials.json to fetch the token to access our public api; under the projects folder we can find the entities to work with; gulpfile.js, gulp-tasks, package.json, and node_modules are required to compile the files with code. Our endpoint only accepts JS and CSS so the extension will compile your TS and SCSS for you before calling the endpoint.

I created some fixtures to copy the file structure from. This is what the fixture for deploy commands looks like:

And we just copy it.

function createFixture() {
    cpSync(FIXTURE_DEPLOY_BASE_DIR, FIXTURE_DEPLOY_DIR, { recursive: true })
}

...

before(createFixture)
before(createCredentialsFile(FIXTURE_DEPLOY_DIR))

2) Modify the experiment

To test that the deploy command correctly modifies the back-end state of the experiment, I decided to randomize the code in JS and CSS files, and some fields in config files. This way we have new values every time we call the test and can check if the back-end state is updates to those values.

function generateCssCode() {
    return `.body {
    width: ${Math.ceil(Math.random() * 100)}px;
}`
}

function generateJsCode() {
    return `console.log(${Math.ceil(Math.random() * 100)});`
}

const experimentVariationJs = generateJsCode()
const experimentVariationCss = generateCssCode()

function modifyEntities() {
    writeFileSync(EXP_VARIATION_JS_PATH, experimentVariationJs)
    writeFileSync(EXP_VARIATION_CSS_PATH, experimentVariationCss)
}

...

// in test case
before(modifyEntities)

I saved the generated code in the file scope to later access it when assessing the results.

3) Open the workspace

We programmatically open the workspace from within the test.

const openFixtureFolder = (fixtureDir: string) => () =>
    browser.executeWorkbench((vscode, folder) => {
       vscode.commands.executeCommand('vscode.openFolder', vscode.Uri.file(folder))
    }, fixtureDir)

before(openFixtureFolder(FIXTURE_DEPLOY_DIR))

This function is reused across multiple test files, hence it’s curried.

4) Wait for the token to be fetched

await hasDisplayedNotification(workbench, `Data for ${FIXTURE_DEPLOY} has been downloaded`)

This is the notification that is displayed by the extension when it fetches the initial data with the credentials provided to it.

5) Install node_modules

we install the node_modules in the workspace to be able to compile the code.

execSync('npm i', { cwd: FIXTURE_DEPLOY_DIR })

6) Run the deploy command

I’m using macOS and the testing frameworks can’t quite use the native context menus. You can open the menu programmatically but you can’t click on the elements.

Here’s the issue about it in the wdio-vscode-service . It’s also in the list known issues of vscode-extension-tester, another popular framework for VSCE testing.

My workaround

Essentially, what calling a command via context menu does is calling your function with an argument based on the file you clicked on. We can emulate it by adding another command to command palette that would call the command you want using commands.executeCommand, using custom input arguments that are set manually in an inputBox.

My implementation of it looks like this:

context.subscriptions.push(commands.registerCommand('Kameleoon.runCommand', runCommand))

async function runCommand() {
    const command = await window.showInputBox({ placeHolder: `enter command` })
    if (!command) return window.showErrorMessage('command is empty')

    const args = JSON.parse((await window.showInputBox({ placeHolder: `enter arguments` })) || '{}') 

    if (process.platform === 'win32' && args.fsPath) {
        args.fsPath = args.fsPath.split('/').join(sep)
    }

    commands.executeCommand(command, args)
}

For Windows, before passing the fsPath argument in the inputBox i remove the Windows separator path.split(sep).join(‘/’) and in the runCommand i return it fsPath.split(‘/’).join(sep) I do this because JSON.parse throws an error when it encounters backslashes.

We don’t want users to see the command in their command palette. So let’s only show it in the testing environment. Wdio automatically sets the NODE_ENV variable to ‘test’, how nice of them! Let’s use it.

const IS_TEST_ENV = process.env.NODE_ENV === 'test'

if (IS_TEST_ENV) {
      commands.executeCommand('setContext', 'Kameleoon.isRunCommandVisible', true)
}

And in your package.json under contributes.menus.commandPalette.

{
    "command": "Kameleoon.runCommand",
    "when": "Kameleoon.isRunCommandVisible"
}

Now we can finally execute the command.

await runDeployExperiment(workbench, EXPERIMENT_PATH)

async function runDeployExperiment(workbench: Workbench, path: string) {
    const inputBox = await workbench.executeCommand('Kameleoon - Run command')
    await inputBox.setText('Kameleoon.deployExperiment')
    await inputBox.confirm()
    await inputBox.setText(`{"fsPath": "${convertWinPath(path)}"}`)
    await inputBox.confirm()
}

function convertWinPath(path: string) { // make windows path json-compatable
    return path.split(sep).join('/')
}

7) Confirm command success

await hasDisplayedNotification(workbench, 'Experiment successfully deployed')

7.5) Delete the experiment files before fetching

Since we modified and deployed them. Let’s remove them before fetching the experiment to be sure that the backend state was modified.

removeModifiedEntities()

function removeModifiedEntities() {
    rmSync(EXP_VARIATION_JS_PATH)
    rmSync(EXP_VARIATION_CSS_PATH)
}

8) Fetch experiment

I use the native extension method to fetch experiment, so we not only test the deploy command but also the fetch command.

await runFetchExperiment(workbench, EXPERIMENT_PATH)
await hasDisplayedNotification(workbench, 'Data downloaded!')

async function runFetchExperiment(workbench: Workbench, path: string) {
    const inputBox = await workbench.executeCommand('Kameleoon - Run command')
    await inputBox.setText('Kameleoon.fetchExperiment')
    await inputBox.confirm()
    await inputBox.setText(`{"fsPath": "${convertWinPath(path)}"}`)
    await inputBox.confirm()
}

9) Assess the fetched experiment

We use the randomly generated JS and CSS to see if the fetched experiment corresponds to them.

const experiment = {
    variationJs: experimentVariationJs,
    variationCss: experimentVariationCss
}

assessExperiment(FIXTURE_DEPLOY_DIR, experiment)

The assessExperiment function is extensive, but essentially we check for the existence of all the files that an experiment should have after fetching it. Ex:

expect(existsSync(variationJsFilePath)).toBeTruthy()

We check the correctness of config file contents.

const info = JSON.parse(String(readFileSync(infoFilePath)))

expect(info).toMatchObject({
    id: expect.any(Number),
    siteCode: expect.any(String) 
    etc ...
})

And we check the code values.

const fetchedCode = String(readFileSync(variationJsFilePath))

expect(fetchedCode).toStrictEqual(experiment.variationJs)

And in the end we delete the workspace and voilà.

after(deleteFixture)

function deleteFixture() {
    rmSync(FIXTURE_DEPLOY_DIR, { recursive: true })
}

Test for correct command palette commands

Another test I’ve added ensures the presence of the correct commands in the command palette. This test helps guarantee that only the intended commands are available in the palette.

It sounds simple, but it isn’t. There is no concise way to obtain all the commands from the command palette. Here’s how I collect the extension-related commands in the test:

async function getAvailableCommands(workbench: Workbench) {
    const availableKameleoonCommands: Set<string> = new Set()


    const inputBox = await workbench.openCommandPrompt()
    await inputBox.setText('>kameleoon - ')


    let quickPicksIndex = 0
    while (true) {
        const picks = await inputBox.getQuickPicks()


        for (const pick of picks) {
            availableKameleoonCommands.add(await pick.getLabel())
        }


        quickPicksIndex += picks.length
        const nextPageFirstElement = await inputBox.findQuickPick(quickPicksIndex)


        if (!nextPageFirstElement) break
    }


    return availableKameleoonCommands
}

All our commands start with ‘Kameleoon — ‘, so I set the input box text accordingly and collect them into a set until there are no more. It’s a bit hacky, but it works.

And the test case itself:

describe('Command palette', () => {
    it('should have the correct Kameleoon commands available in command palette', async  () => {
        const workbench = await browser.getWorkbench()
        const availableKameleoonCommands = await getAvailableCommands(workbench)
        expect(availableKameleoonCommands).toEqual(COMMAND_PALETTE_COMMANDS)
    })
})

In COMMAND_PALETTE_COMMANDS I have listed all the commands I want to be displayed in the command palette. So now, even if you forget to specify command palette availability in contributes.menus.commandPalette in package.json, this test will remind you to do so!

The what

In this blog post I’ve only talked about 2 tests, but in our extension we have 6 test files, testing 24 commands in total. For every file wdio creates a separate VS Code instance so the tests don’t take much time because they’re running at the same time.

50 seconds, nifty! Manual testing would take so much longer

As such, the integration tests addressed the challenges of a limited user base and the need for manual testing. These tests have not only improved the extension’s reliability but also streamlined the development and release process.

About Kameleoon

Kameleoon empowers brands to build better products and digital experiences. It is the only optimization solution with Web Experimentation, Feature Experimentation, and AI-Driven Personalization capabilities in a single unified platform. Designed to pull all teams together, Kameleoon supports both product and marketing-led teams to increase their experimentation velocity and leverage their tech stacks. HIPAA, GDPR, and CCPA compliant, Kameleoon is already used by over 1000 medium and enterprise-sized companies to increase visitor engagement and power growth.

Behind Kameleoon is a community of top-tier engineers and developers. Together, they create innovative features and develop the tools for web and feature experimentation of tomorrow. They share the outcomes of their research and their insights on their latest innovations on Medium.