Reading: PRN — Progressive Rethinking Network (HEVC Filtering)

Outperforms VRCNN, DCAD and DRN, 10.1% BD-Rate Reduction Compared to HEVC Baseline

In this story, Partition Tree Guided Progressive Rethinking Network for in-Loop Filtering of HEVC (PRN), by Peking University, is presented. I read this because I work on video coding research. In this paper:

• A Progressive Rethinking Block (PRB) is proposed to deliver long-term memory between the neighboring blocks and allow information to flow without compression.
• Progressive Rethinking Network (PRN) is established by PRBs.
• Multi-scale Mean value of Coding Units (MM-CU) to generate the side information maps which guide the training.

This is a paper in 2019 ICIP. (Sik-Ho Tsang @ Medium)

Outline

1. Progressive Rethinking Block (PRB)
2. PRN: Network Architecture
3. MM-CU Generation and Fusion
4. Experimental Results
5. Experimental Results on GitHub

1. Progressive Rethinking Block (PRB)

Progressive Rethinking Block (PRB)

• For the k-th PRB, it receives feature maps Fk as a common input to generate multi-level feature maps Gk.

• Mk works as a long-term memory which is also generated by the previous PRB. Mk is concatenated with Gk.
• That is the orange line in the above figure, it is also known as inter-block skip-connection.
• After that, two 1×1 convolutional layers are used to respectively extract two new feature maps Mk+1 and Fk+1, denoted by PM(.) and PF(.) as follows:

• where [.] is the concatenation.
• M0 = F0 as the initial.

2. PRN: Network Architecture

PRN: Network Architecture

• First, there is a Low-level Feature Extractor (LFE). The unfiltered frame x first is fed into an LFE consisting of two convolutional layers to extract shallow feature F0 which is to be fed into sequential PRBs:

• Then, F0 goes through stacked multiple PRBs:

• After generating FD (D is the number of PRBs), all feature maps F0, F1, …, FD, are concatenated together and convolved by a 1×1 convolutional layer, denoted by PCompress(.), in order to compress them as follows:

• Next, a global residual is used, which facilitates better detail modeling and network training. After two convolutional layers, the output frame is finally reconstructed.
• 10 PRBs is used.
• In each PRB, the multi-level feature map is generated by 6 convolutional layers with dense connection.

3. MM-CU Generation and Fusion

MM-CU Generation

• The mean value of a CU everytime a partition happens.
• This side information can guide the network to remove the coding artifacts at different scales according to the entire coding partition architecture.

The detailed structure of the SIFE unit.

• A shallow CNN is used as a feature extractor to extract feature maps from them.
• The architecture of the extractor is similar to PRN but much shallower.
• Finer M-CU maps contain more local details of the coding architecture while coarser ones contain some global coding structure information.
• Coarser M-CU maps are fused with the main branch in deeper layers.
• The fusion is simply using element-wise add.
• The feature maps of MM-CU are added after the 2-nd, 4-th, 6-th, 8-th PRB of the PRN baseline.

4. Experimental Results

4.1. Training

• DIV2K is used for training.
• MSE is used as loss function.
• HM-16.15 is used.

4.2. BD-Rate

BD-Rate (%) Under AI Configuration

• 10.1% BD-rate reduction is obtained.

5. Experimental Results on GitHub

5.1. BD-Rate

BD-Rate (%) Under AI Configuration

• To verify the inter-block skip-connection and the guidence of MM-CU, ablation study is performed as above.
• With inter-block skip-connection, i.e. PRN, it outperforms RDN.
• With also the MM-CU, i.e. PRN-M, it outperforms the PRN baseline.

5.2. SOTA Comparison

BD-Rate (%) Under AI Configuration

• PRN outperforms VRCNN, DCAD and DRN.

This is the 6th story in this month.

References

[2019 ICIP] [PRN] Partition Tree Guided Progressive Rethinking Network for in-Loop Filtering of HEVC Corresponding GitHub Corresponding Poster

Codec Filtering

JPEG [ARCNN] [RED-Net] [DnCNN] [Li ICME’17] [MemNet] [MWCNN] HEVC [Lin DCC’16] [IFCNN] [VRCNN] [DCAD] [MMS-net] [DRN] [Lee ICCE’18] [DS-CNN] [CNNF] [RHCNN] [VRCNN-ext] [S-CNN & C-CNN] [MLSDRN] [ARTN] [Double-Input CNN] [CNNIF & CNNMC] [B-DRRN] [Residual-VRN] [Liu PCS’19] [DIA_Net] [RRCNN] [QE-CNN] [Jia TIP’19] [EDCNN] [VRCNN-BN] [MACNN] 3D-HEVC [RSVE+POST] AVS3 [Lin PCS’19] [CNNLF] VVC [AResNet] [Lu CVPRW’19] [Wang APSIPA ASC’19] [ADCNN] [PRN] [DRCNN] [Zhang ICME’20] [MGNLF]

My Other Previous Readings