Add multi langauge support using ffi and cdp.

[objecthuman]

Problem

The library is written in Golang, and let's be honest here, nobody uses Golang for web automation or web testing. Most developers out there either use Node.js or Python. Since people will want to use the Python version, we will have to write the Python port. Writing locatr from scratch in Python/Node.js is time-consuming; we don't have enough people to maintain the package, and if we want to make a change, then we will have to change all the x number of ports, which will be a lot of pain.

Solution

Remove dependency on playwright and use CDP.

• The first thing that we have to do is remove our dependency on packages like playwright. Instead of using Playwright, we will directly use CDP to interact with the web page.
• In Golang, there is an actively maintained CDP package that has all the features we need and is type-safe. So, to interact with CDP, we can use this package.
• The end user will have to open the CDP session and provide us with the targetId (a unique identifier for the page). After we have the targetId, we can do anything on the page.

Solution for creating multiple pacakge ports.

I have thought of multiple ways of tackling this issue, but I will be going with using FFI. Here are some of the solutions I thought of initially:

1. Expose a grpc/http server and make the Python client consume it: I don't want to start an http/grpc server because the user will have to specify the port for the server and additional configurations. Also running another server will use more cpu/ram.

2. Use inter-process communication: We could go this route, but then we would have to handle multiple processes and send data through them. This also comes with a cost of being harder to debug.

3. Use Golang's Foreign Function Interface: Since Golang has features to compile itself into a C binary, we can expose the Golang functions to the binary, which we can use with the Ctypes package in Python and ffi package in Node.js. Below is an example of how we can do this.
   main.go

package main

import (
	"C"
	"context"
	"log"
	"math/rand"
	_ "unsafe"

	"github.com/mafredri/cdp"
	"github.com/mafredri/cdp/protocol/runtime"
	"github.com/mafredri/cdp/rpcc"
)
import "fmt"

var ConnectionMap = make(map[int8]*rpcc.Conn)

func CreateRpcConn(port int, pageId string) int8 {
	ctx := context.Background()
	wsUrl := fmt.Sprintf("ws://localhost:%d/devtools/page/%s", port, createString(pageId))
	conn, err := rpcc.DialContext(ctx, wsUrl)
	if err != nil {
		log.Println("failed to dail context", err)
		return int8(-1)
	}
	randInt := int8(rand.Intn(255))
	ConnectionMap[randInt] = conn
	fmt.Println("The connection id in golang is", randInt)
	return int8(randInt)
}

Compile the file with: go build -o a.out -buildmode=c-shared examples/cdp/main.go.
Call CreateRpcConn in python: main.py

import ctypes

mylib = ctypes.CDLL("./a.out")

port = 5002

page_id = "4BD749CA824CECE7D48B906D79C754F3"

mylib.CreateRpcConn.argtypes = [ctypes.c_int, ctypes.c_char_p]
mylib.CreateRpcConn.restype = ctypes.c_int8

port = ctypes.c_int(port)
page_id = page_id.encode("utf-8")

connection_id = mylib.CreateRpcConn(port, page_id)

print("Connection ID in python:", ctypes.c_int8(connection_id))

Then we will just expose some helper functions from python, nodejs for the end user to interact with the pacakge.