Synthetic vision

Synthetic vision is achieved in CoppeliaSim via vision sensors: they can perform offscreen rendering of the scene in different modes. The generated image can then be modified/processed in various ways, for instance from within a vision callback function:

image processing with OpenCV (via the image plugin)

simple image processing via the vision plugin

sim.transformImage

sim.transformBuffer

Following shows a vision callback function that illustrates 3 different ways to modify an acquired image:


function sysCall_vision(inData)
    -- inData.handle: handle of the vision sensor
    -- inData.resolution: resolution of the vision sensor
    -- inData.clippingPlanes: near and far clipping plane of the vision sensor
    -- inData.viewAngle: the view angle if the vision sensor is in perspective operation
    -- inData.orthoSize: the size of the view if the vision sensor is in orthogonal operation
    -- inData.perspectiveOperation: whether the vision sensor is in perspective operation

    -- e.g. use very simple functions from the vision plugin (simExtVision):
    simVision.sensorImgToWorkImg(inData.handle)
    simVision.edgeDetectionOnWorkImg(inData.handle,0.2)
    simVision.swapWorkImgWithBuffer1(inData.handle)
    simVision.uniformImgToWorkImg(inData.handle,{0.5,0.25,0})
    simVision.addBuffer1ToWorkImg(inData.handle)
    simVision.workImgToSensorImg(inData.handle)
    
    -- and/or, use more powerful functions from the image plugin (simExtImage):
    if simIM then
        local imgHandle=simIM.readFromVisionSensor(inData.handle)
        local center={inData.resolution[1]/2,inData.resolution[2]/2}
        local radius=(inData.resolution[1]+inData.resolution[2])/8
        simIM.circle(imgHandle,center,radius,{255,255,0},4)
        simIM.writeToVisionSensor(imgHandle,inData.handle)
        simIM.destroy(imgHandle)
    end

    -- Or you can directly operate on the image buffer:
    local image=sim.getVisionSensorImg(inData.handle)
    sim.transformImage(image,inData.resolution,4)
    image=sim.transformBuffer(image,sim.buffer_uint8rgb,1,0,sim.buffer_uint8bgr)
    sim.setVisionSensorImg(inData.handle,image)
    
    outData={}
    outData.trigger=true -- whether the sensor should trigger
    -- filters may append packets (in packed form, use sim.packFloatTable to pack) to this table:
    outData.packedPackets={sim.packFloatTable({1,42,57})} 
    return outData
end