Building a Collaborative Go Sandbox: Part 2 (Collaborative Editing)

pretty picture

· Basic Setup · yjs WebSocket Server Changes · Frontend Implementation ∘ Terminology ∘ Basic Collaborative Doc ∘ Adding Multiple File Support ∘ Terminal · Conclusion

In part 1 I went over how I built a remote execution server for executing Go code. Now is the exciting part — building the collaborative editing! And for that I’m going to be using yjs with Monaco editor (the same editor that VS Code uses!) GitHub repo

This is what I ended up with:

Demo of the final project

Basic Setup

For this project I originally tried using the WebRTC Provider, but I ran into an issue. I wanted the user to load the Go files from the backend, but only if they weren’t loaded by someone else. The only way I got it to work was with a hacky call to setTimeout — if no data was loaded after some time, then make a request to the backend to fetch the files — but this wasn’t a satisfying approach. Instead, I opted to use WebSocket Provider. This way, all the document updates happen via a centralised server with WebSockets. I used the official yjs WebSocket server starter project to facilitate this.

This does mean there are 2 separate “backend” services, the Go backend service for remote execution I built in part 1, and the Node.js WebSocket server for yjs document updates.

Service Setup

This setup allows me to load and persist files, without cutting corners and hacks. I decided to write the files to the Go backend only when the last client disconnects — this way we can avoid expensive I/O operations unnecessarily during the active editing of files.

The WebSocket provider already allows us to specify the room name, and the current session-id is a really fitting value.

yjs WebSocket Server Changes

The dev server provided a function called setPersistence , which I called directly in the server.js file:

setPersistence({
  bindState: async (docName, ydoc) => {
    const filesArray = ydoc.getArray("files");
    const state = ydoc.getMap("state");

    try {
      const data = await fetchSessionFiles(docName);

      const fileNames = Object.keys(data);

      filesArray.push(fileNames);
      fileNames.forEach((name) => {
        const yText = ydoc.getText(name);
        yText.insert(0, data[name]);
      });

      state.set("activeFile", fileNames[0]);
    } catch (error) {
      console.error(error);
      state.set("error", "true");
    }
  },
  writeState: async (docName, ydoc) => {
    try {
      const filesArray = ydoc.getArray("files");
      const fileMap = /** @type { Record<string,string> } */ ({});

      for (const fileName of filesArray.toArray()) {
        const yText = ydoc.getText(fileName);
        fileMap[fileName] = yText.toString();
        console.log({ [fileName]: yText.toString() });
      }

      await saveSessionFiles(docName, fileMap);

      console.log("writeState: " + docName);
    } catch (error) {
      console.error(error);
    }
  },
});

Where fetchSessionFiles and saveSessionFiles are simple functions that wrap a call to fetch the appropriate endpoints in the Go server.

Here the docName is the current session-id , and specific files are retrieved with ydoc.getText(fileName) .

Frontend Implementation

Terminology

In Monaco editor:

• Model: content — represents a file that was opened
• Editor: facing view of the model

yjs is an implementation of CRDT (Conflict-Free Replicated Data Type) that’s really popular. For yjs:

• Connection provider: facilitates communication between users
• Editor binding: connects data to an editor

Basic Collaborative Doc

We can follow the yjs tutorial for a simple setup. I am using Next.js for the frontend, so I had to alter it slightly. After setting up a Next.js project, I installed these dependencies with: npm install yjs monaco-editor y-monaco y-websocket

• monaco-editor : the Monaco text editor itself
• yjs : a popular CRDT implementation, the core of the collaborative editor
• y-monaco : Editor bindings for binding the state of Monaco editor to a syncable Yjs document
• y-websocket : provider that connects clients directly with each other using WebSockets

Bringing them all together:

"use client";
import { Doc } from "yjs";
import { WebsocketProvider } from "y-websocket";
import { MonacoBinding } from "y-monaco";
import { useEffect, useRef } from "react";
import { editor } from "monaco-editor";

export default function Home() {
  const yDocRef = useRef<Doc>(null);

  useEffect(() => {
    if (yDocRef.current) return; // don't recreate the Doc
    yDocRef.current = new Doc();
    const provider = new WebsocketProvider(
      "ws://localhost:1234",
      sessionId,
      yDocRef.current,
    );
    const type = yDocRef.current.getText("monaco");
    const monacoEditorDiv = document.getElementById("monaco-editor");
    if (!monacoEditorDiv) return;
    const monacoEditor = editor.create(monacoEditorDiv, {
      value: "",
      language: "go",
      theme: "vs-dark",
    });
    const model = monacoEditor.getModel();
    if (!model) return;
    const monacoBinding = new MonacoBinding(
      type,
      model,
      new Set([monacoEditor]),
      provider.awareness,
    );
  }, []);

  return (
    <div className="min-h-screen">
      <div id="monaco-editor" className="h-full min-h-screen"></div>
    </div>
  );
}

Here I use a ref to ensure I only have 1 instance of the yjs document at a time.

Adding Multiple File Support

The answer to how to actually switch the content for a file comes from this thread, where a user responds:

y-monaco binds a Y.Text to an ITextModel… You can simply switch to a different ITextModel and render that without losing history from other documents

So now we need to do 2 things:

• keep track of the current file that is opened
• keep track of Y.Text, ITextModel and Bindings for all the files, based on the file name

We can keep track of the files with an Array type, and track active file with a state Map type. In yjs, we can leverage the yjs Array type for tracking the file names, and Map type for tracking the state (currently only the active file).

export default function CodeEditor({
  ref,
  codeRef,
  sessionId = "",
}: {
  ref: React.Ref<CodeEditorHandle>;
  codeRef: React.Ref<CodeExecutor>;
  sessionId?: string;
}) {
  const yDocRef = useRef<Doc>(null);
  const providerRef = useRef<WebsocketProvider>(null);
  const bindingsRef = useRef<Map<string, MonacoBinding>>(new Map());
  const modelsRef = useRef<Map<string, editor.ITextModel>>(new Map());
  const editorRef = useRef<editor.IStandaloneCodeEditor | null>(null);

  const [files, setFiles] = useState<string[]>([]);
  const [activeFile, setActiveFile] = useState<string>();

  useImperativeHandle(ref, () => ({
    getValue: () => editorRef.current?.getValue() ?? "",
  }));

  useImperativeHandle(codeRef, () => ({
    getCodeMap: () => {
      const result: { [key: string]: string } = {};
      modelsRef.current.forEach((value, key) => {
        result[key] = value.getValue();
      });
      return result;
    },
  }));

  function setSharedActiveFile(name: string) {
    if (yDocRef.current) {
      yDocRef.current.getMap("state").set("activeFile", name);
    } else {
      setActiveFile(name);
    }
  }

  function handleAddFile(name: string) {
    const yFiles = yDocRef.current?.getArray<string>("files");
    if (!yFiles || yFiles.toArray().includes(name)) return;
    yFiles.push([name]);
    setSharedActiveFile(name);
  }

  function handleDeleteFile(name: string) {
    const yFiles = yDocRef.current?.getArray<string>("files");
    if (yFiles) {
      const idx = yFiles.toArray().indexOf(name);
      if (idx !== -1) yFiles.delete(idx, 1);
    }

    bindingsRef.current.get(name)?.destroy();
    bindingsRef.current.delete(name);
    modelsRef.current.get(name)?.dispose();
    modelsRef.current.delete(name);

    if (activeFile === name) {
      const remaining =
        yDocRef.current?.getArray<string>("files").toArray() ?? [];
      if (remaining.length > 0) {
        setSharedActiveFile(remaining[0]);
      } else {
        editorRef.current?.setModel(null);
        setActiveFile(undefined);
      }
    }
  }

  const createModelBinding = useCallback((activeFile: string) => {
    const model = editor.createModel("", "go");
    const yText = yDocRef.current!.getText(activeFile);
    const binding = new MonacoBinding(
      yText,
      model,
      new Set([editorRef.current!]),
      providerRef.current!.awareness,
    );
    modelsRef.current.set(activeFile, model);
    bindingsRef.current.set(activeFile, binding);
  }, []);

  useEffect(() => {
    const monacoEditorDiv = document.getElementById("monaco-editor");
    if (!monacoEditorDiv) return;

    if (!yDocRef.current) {
      yDocRef.current = new Doc();
      providerRef.current = new WebsocketProvider(
        "ws://localhost:1234",
        sessionId,
        yDocRef.current,
      );
      editorRef.current = editor.create(monacoEditorDiv, {
        value: "",
        language: "go",
        theme: "vs-dark",
      });

      const yFiles = yDocRef.current.getArray<string>("files");
      const yState = yDocRef.current.getMap("state");

      yFiles.observe(() => {
        const updatedFileNames = yFiles.toArray();
        setFiles(updatedFileNames);
        // create models and bindings for each file name

        // filter out the new ones
        const newFileNames = updatedFileNames.filter(
          (updatedFileName) => !modelsRef.current.has(updatedFileName),
        );
        // create models for each
        newFileNames.forEach(createModelBinding);
      });
      yState.observe(() => {
        const af = yState.get("activeFile") as string | undefined;
        if (af) setActiveFile(af);
      });

      setFiles(yFiles.toArray());
      const af = yState.get("activeFile") as string | undefined;
      if (af) setActiveFile(af);
    }

    return () => {
      // tear everything down
      // provider, bindings, models, yDoc
      bindingsRef.current.forEach((binging) => binging.destroy());
      modelsRef.current.forEach((model) => model.dispose());
      editorRef.current?.dispose();
      providerRef.current?.disconnect();
      providerRef.current?.destroy();
      yDocRef.current?.destroy();

      bindingsRef.current = new Map();
      modelsRef.current = new Map();
      editorRef.current = null;
      providerRef.current = null;
      yDocRef.current = null;

      setActiveFile(undefined);
      setFiles([]);
    };
  }, [sessionId, createModelBinding]);

  useEffect(() => {
    if (!activeFile) return;

    // if the user created a new file
    if (!modelsRef.current.has(activeFile)) {
      createModelBinding(activeFile);
    }

    editorRef.current!.setModel(modelsRef.current.get(activeFile)!);
  }, [activeFile, createModelBinding]);

  return (
    <div className="flex flex-1 min-h-0">
      <FileSystem
        files={files}
        activeFile={activeFile}
        onFileSelect={setSharedActiveFile}
        onAddFile={handleAddFile}
        onDeleteFile={(name) => handleDeleteFile(name)}
      />
      <div id="monaco-editor" className="flex-1 min-h-0" />
    </div>
  );
}

Lets break this down. We store bindings and models with:

const yDocRef = useRef<Doc>(null);
const providerRef = useRef<WebsocketProvider>(null);
const bindingsRef = useRef<Map<string, MonacoBinding>>(new Map());
const modelsRef = useRef<Map<string, editor.ITextModel>>(new Map());
const editorRef = useRef<editor.IStandaloneCodeEditor | null>(null);

By using maps for bindings and models, we can easily swap out the Y.Text to an ITextModel as this thread suggests.

We also need to store the file names and the active file in the component state, so we can render it:

const [files, setFiles] = useState<string[]>([]);
const [activeFile, setActiveFile] = useState<string>();

Then there’s 3 functions:

function setSharedActiveFile(name: string) {
    // omitted for brevity
  }

  function handleAddFile(name: string) {
    // omitted for brevity
  }

  function handleDeleteFile(name: string) {
    // omitted for brevity
  }

They’re just helper functions to manage the files representation.

The most important function for file switching is this:

const createModelBinding = useCallback((activeFile: string) => {
  const model = editor.createModel("", "go");
  const yText = yDocRef.current!.getText(activeFile);
  const binding = new MonacoBinding(
    yText,
    model,
    new Set([editorRef.current!]),
    providerRef.current!.awareness,
  );
  modelsRef.current.set(activeFile, model);
  bindingsRef.current.set(activeFile, binding);
}, []);

This function actually creates the binding and the model, and updates modelsRef and bindingsRef maps accordingly.

useEffect(() => {
  const monacoEditorDiv = document.getElementById("monaco-editor");
  if (!monacoEditorDiv) return;

  if (!yDocRef.current) {
    yDocRef.current = new Doc();
    providerRef.current = new WebsocketProvider(
      "ws://localhost:1234",
      sessionId,
      yDocRef.current,
    );
    editorRef.current = editor.create(monacoEditorDiv, {
      value: "",
      language: "go",
      theme: "vs-dark",
    });

    const yFiles = yDocRef.current.getArray<string>("files");
    const yState = yDocRef.current.getMap("state");

    yFiles.observe(() => {
      const updatedFileNames = yFiles.toArray();
      setFiles(updatedFileNames);
      // create models and bindings for each file name

      // filter out the new ones
      const newFileNames = updatedFileNames.filter(
        (updatedFileName) => !modelsRef.current.has(updatedFileName),
      );
      // create models for each
      newFileNames.forEach(createModelBinding);
    });
    yState.observe(() => {
      const af = yState.get("activeFile") as string | undefined;
      if (af) setActiveFile(af);
    });

    setFiles(yFiles.toArray());
    const af = yState.get("activeFile") as string | undefined;
    if (af) setActiveFile(af);
  }

  return () => {
    // tear everything down
    // provider, bindings, models, yDoc
    bindingsRef.current.forEach((binging) => binging.destroy());
    modelsRef.current.forEach((model) => model.dispose());
    editorRef.current?.dispose();
    providerRef.current?.disconnect();
    providerRef.current?.destroy();
    yDocRef.current?.destroy();

    bindingsRef.current = new Map();
    modelsRef.current = new Map();
    editorRef.current = null;
    providerRef.current = null;
    yDocRef.current = null;

    setActiveFile(undefined);
    setFiles([]);
  };
}, [sessionId, createModelBinding]);

This useEffect sets up the yjs document using the WebSocket Provider, pointing to our yjs WebSocket server. It also observes the “files” array and “state” map for changes, updating the component state accordingly, and creating new models and bindings if a new file is added to the yjs document. This has to run whenever the sessionId changes, and cleans up all the resources on unmount to prevent leaks.

useEffect(() => {
  if (!activeFile) return;

  // if the user created a new file
  if (!modelsRef.current.has(activeFile)) {
    createModelBinding(activeFile);
  }

  editorRef.current!.setModel(modelsRef.current.get(activeFile)!);
}, [activeFile, createModelBinding]);

This useEffect simply swaps out the models and bindings, whenever the active file is changed.

return (
  <div className="flex flex-1 min-h-0">
    <FileSystem
      files={files}
      activeFile={activeFile}
      onFileSelect={setSharedActiveFile}
      onAddFile={handleAddFile}
      onDeleteFile={(name) => handleDeleteFile(name)}
    />
    <div id="monaco-editor" className="flex-1 min-h-0" />
  </div>
);

Then we simply render the files with a basic FileSystem component, which just renders all the files in a pretty way and attaches some handlers.

Terminal

In order to display the stdout and stderr results of running the Go code in the frontend, I used WebSockets. I used the same approach as with another project I built (a basic WebSocket chat app, GitHub repo). Except the only difference is that the clients never write, they only ever read, and they’re grouped by session.

In the Go backend, I added another endpoint /terminal , and implemented a basic component that opens a WebSocket connection and displays the results. Then, in the run function, instead of printing the stdout and stderr , I simply write the results to the appropriate WebSocket connections.

With the terminal complete, this finishes the project!

Conclusion

This was an incredibly fun project to build, with a couple of services, and a few moving parts. I arrived at a solution with which I’m content — I have a collaborative Go sandbox, which can be completely self-hosted.