Synthetic vision

Synthetic vision is achieved in CoppeliaSim via vision sensors: they can perform offscreen rendering of the scene in different modes. The generated image can then be modified/processed in various ways, for instance from within a vision callback function:

image processing with OpenCV (via the image plugin)

simple image processing via the vision plugin

sim.transformImage

sim.transformBuffer

Following shows a vision callback function that illustrates 3 different ways to modify an acquired image:

#python

def sysCall_init():
    sim = require('sim')
    simVision = require('simVision')
    simIM = require('simIM')
    
def sysCall_vision(inData):
    # inData['handle']: handle of the vision sensor
    # inData['resolution']: resolution of the vision sensor
    # inData['clippingPlanes']: near and far clipping plane of the vision sensor
    # inData['viewAngle']: the view angle if the vision sensor is in perspective operation
    # inData['orthoSize']: the size of the view if the vision sensor is in orthogonal operation
    # inData['perspectiveOperation']: whether the vision sensor is in perspective operation

    # e.g. use very simple functions from the vision plugin (simVision):
    simVision.sensorImgToWorkImg(inData['handle'])
    simVision.edgeDetectionOnWorkImg(inData['handle'], 0.2)
    simVision.swapWorkImgWithBuffer1(inData['handle'])
    simVision.uniformImgToWorkImg(inData['handle'], [0.5, 0.25, 0])
    simVision.addBuffer1ToWorkImg(inData['handle'])
    simVision.workImgToSensorImg(inData['handle'])

    # and/or, use more powerful functions from the image plugin (simImage):
    imgHandle = simIM.readFromVisionSensor(inData['handle'])
    center = [inData['resolution'][0] / 2, inData['resolution'][1] / 2]
    radius = (inData['resolution'][0] + inData['resolution'][1]) / 8
    simIM.circle(imgHandle, center, radius, [255, 255, 0], 4)
    simIM.writeToVisionSensor(imgHandle, inData['handle'])
    simIM.destroy(imgHandle)

    # Or you can directly operate on the image buffer:
    image = sim.getVisionSensorImg(inData['handle'])
    sim.transformImage(image, inData['resolution'], 4)
    image = sim.transformBuffer(image, sim.buffer_uint8rgb, 1, 0, sim.buffer_uint8bgr)
    sim.setVisionSensorImg(inData['handle'], image)

    outData = {}
    outData['trigger'] = True  # whether the sensor should trigger
    # filters may append packets (in packed form, use sim.packFloatTable to pack) to this table:
    outData['packedPackets'] = [sim.packFloatTable([1, 42, 57])]
    return outData

--lua

function sysCall_init()
    sim = require('sim')
    simVision = require('simVision')
    simIM = require('simIM')
end

function sysCall_vision(inData)
    -- inData.handle: handle of the vision sensor
    -- inData.resolution: resolution of the vision sensor
    -- inData.clippingPlanes: near and far clipping plane of the vision sensor
    -- inData.viewAngle: the view angle if the vision sensor is in perspective operation
    -- inData.orthoSize: the size of the view if the vision sensor is in orthogonal operation
    -- inData.perspectiveOperation: whether the vision sensor is in perspective operation

    -- e.g. use very simple functions from the vision plugin (simVision):
    simVision.sensorImgToWorkImg(inData.handle)
    simVision.edgeDetectionOnWorkImg(inData.handle, 0.2)
    simVision.swapWorkImgWithBuffer1(inData.handle)
    simVision.uniformImgToWorkImg(inData.handle, {0.5, 0.25, 0})
    simVision.addBuffer1ToWorkImg(inData.handle)
    simVision.workImgToSensorImg(inData.handle)
    
    -- and/or, use more powerful functions from the image plugin (simIM):
    local imgHandle = simIM.readFromVisionSensor(inData.handle)
    local center = {inData.resolution[1] / 2, inData.resolution[2] / 2}
    local radius = (inData.resolution[1] + inData.resolution[2]) / 8
    simIM.circle(imgHandle, center, radius, {255, 255, 0}, 4)
    simIM.writeToVisionSensor(imgHandle, inData.handle)
    simIM.destroy(imgHandle)

    -- Or you can directly operate on the image buffer:
    local image = sim.getVisionSensorImg(inData.handle)
    sim.transformImage(image, inData.resolution, 4)
    image = sim.transformBuffer(image, sim.buffer_uint8rgb, 1, 0, sim.buffer_uint8bgr)
    sim.setVisionSensorImg(inData.handle, image)
    
    outData = {}
    outData.trigger = true -- whether the sensor should trigger
    -- filters may append packets (in packed form, use sim.packFloatTable to pack) to this table:
    outData.packedPackets = {sim.packFloatTable({1, 42, 57})} 
    return outData
end