Overview

This project investigates whether Wi-Fi beamforming feedback matrices, or BFMs, can support real-time, camera-free human pose sensing. The system captures compressed BFM reports exchanged between a multi-antenna Wi-Fi access point and a mobile device. Cameras are used only to obtain ground-truth skeleton labels during data collection and training. Real-time inference relies entirely on Wi-Fi BFM data.

System pipeline

  1. BFM collection: The system captures compressed beamforming reports from standard Wi-Fi communication.
  2. Signal reconstruction: Quantized feedback parameters are decompressed into an approximate beamforming representation across 234 subcarriers.
  3. Spatial processing: The MUSIC algorithm converts each BFM packet into a 181-bin angular spectrum.
  4. Temporal inference: A neural network combining a 1D CNN, a time-aware BiLSTM, and an attention mechanism processes a 197-packet sequence.
  5. Pose estimation: The current prototype estimates the neck, left shoulder, left elbow, and left wrist.
  6. Edge deployment: The complete real-time pipeline runs on an NVIDIA Jetson Orin Nano.

Research goal

The research goal is to transform the Wi-Fi link of a single unmodified mobile device into a continuous, privacy-conscious sensor of in-person social interaction. By combining plaintext 802.11ac/ax beamforming feedback with the device’s IMU, the system aims to separate the user’s self-motion from perturbations caused by nearby device-free people. It seeks to recover coarse social signals, including when encounters begin and end, whether another person approaches, departs, or lingers, how many distinct people the user meets, and the direction in which encounters occur. The system is designed to require no camera, extra radio, firmware modification, or cooperation from nearby people, while deliberately excluding identity, faces, and speech.