For the problems of low decoding accuracy, long decoding time, and low quality of decoded video image in the traditional lossless decoding method of compressed coded video, a lossless decoding method of compressed coded video based on inter frame difference background model is proposed. At the coding end, the inter frame difference background model is used to extract the single frame video image, and the mixed coding method is used to compress the video losslessly. At the decoding side, the CS-SOMP (compressive sensing-synchronous orthogonal matching pursuit algorithm) joint reconstruction algorithm is composed of synchronous orthogonal matching pursuit algorithm (SOMP) and K-SVD (kernel singular value decomposition) algorithm to losslessly decode the compressed encoded video. The simulation results show that the lossless decoding method based on the inter frame difference background model has higher accuracy, shorter decoding time, and ensures the quality of the decoded video image.