dayong.ren

Authors：Xinlong Wang / Chunhua Shen / Jiaya Jia
Abstract
- real scene / precisely and intuitively / diversified elements / informative 3D scene
- instance segmentation benefit from semantic segmentation through learning semantic-aware point-level instance embedding
- semantic features of the points belonging to the same instances are fused together to make more accurate per-point
- http://github.com/WXinlong/ASIS
1.Introduction
- be parsed to / panoptic segmentation / raw point / step-wise paradigm / sub-optimal and inefficient / tailor
- The former one distinguishes different instances of the same class clearly， while the latter one wants them to have the same label.
- two sub-optimal approachs: (1) 获取语义分割的结果，在每个语义分割获取的类上进行实例分割 (2) 获取实例分割的结果，并对实例分割结果进行识别，确认不同的实例应该属于哪一类
- ASIS (Associatively Segmenting Instances and Semantica)
2.Related Work
- Instance Segmentation
  - inspired by R-CNN, segment instances by proposing segment candidates
  - based on bounding box proposals
  - Mask R-CNN
  - using the learned associative embedding
  - discriminative loss function which enables to learn the pixel-level instance embedding efficiently
  - sub-grouping problems
  - similarity matrix of a point cloud SGPN
- Semantic Segementation
  - 3D-FCNN
  - Pointnet
  - SPG
- Deep Learning on Point Clouds
  - multiview rendering images part of contextual information in point cloud is left behind during the projection process
  - volumetric representations both computationally and memory intensive due to the sparsity of point clouds and the heavy computation of 3D convolutions
  - raw point PointNet/PointNet++/RSNet/DGCNN/PointCNN
3.Our Method
- 3.1.A Simple Baseline
  - a shaed encoder two parallel decoders
  - Point cloud —>feature encoder—>feature matrix N_P—>
    - semantic segmentation—>decode the feature martix to N_PN_F—>feature matrix F_SEM—>semantic predictions P_SEM N_PN_C
    - instance segmentation—>decode the feature martix to N_PN_F—>feature matrix F_INS—>predict per-point instance embedding E_INS N_PN_E
    - Loss Function is tailored from [6]
- 3.2.Mutual Aid
  - Semantic-aware Instance Segmentation
    - F_SINS = F_INS + FC(F_SEM)
    - 将突出的语义特征加入到实例分割的特征矩阵中，使得矩阵对图像的语义特征敏感
  - Instance-fused Semantic Segmentation
    - K nearest neighbor
    - channel-wise max aggregation
    - 利用K近邻算法，将原始数据中对不同类的隶属关系(亲切程度)通过N_P*K表现出来，然后跟语义分割数据放在一起形成三维特征空间，并根据信道最大聚合，将三维特征空间中的最大隶属度提取出来，进行语义分割。
4.Experiments
- 4.1.Experiment Settings
  - Darasets
    - S3DIS
      - 9-dim feature vector XYZ，RGB，normalized coordinates as to the room
      - 6 areas 272 room 13 categories
      - 3D scans Matterport Scanners
    - ShapeNet
      - 3-dim XYZ
      - 16881 16 categories
      - 3D shapes from 16 categories
  - Evaluation Metrics
    - Semantic segmentation
      - over-all accuracy OACC
      - mean accurac mACC
      - mean IoU mIoU
    - Instance segmentation
      - CoV
      - WcoV
      - mean precision mPrec
      - mean recall mRec
    - Training and Inference Details
      - vanilla
      - RePr
- 4.2.S3DIS Results
  - 4.2.1.Baseline Method
    - efffective and efficient
  - 4.2.2.ASIS
    - the impact of the ASIS on evaluation metrics are analysised
  - 4.2.3.Analsis
    - Ablative Analysis
    - Category-based Analysis
  - 4.2.4.Qualitative Results
  - 4.2.5.Computation Time
- 4.3.ShapeNet Results
5.Conclusion
- hoping the novel design provides insights to future works on segmentation tasks panoptic segmentation and beyondRe
References
- [6] Semantic instance segmentation with a discriminative loss function

Associatively Segmenting Instances and Semantics in Point Clouds