Visual Effects
-- Using nerfies you can create fun visual effects. This Dolly zoom effect - would be impossible without nerfies since it would require going through a wall. -
- -
-
@@ -123,78 +94,17 @@
-
-
-
-
-
-
- Nerfies: Deformable Neural Radiance Fields
+✨NeuralFuse✨
+Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes
-
-
-
-
-
-
- Paper
-
-
-
@@ -82,19 +72,9 @@
arXiv
-
-
-
-
-
-
- Video
-
-
-
@@ -102,15 +82,6 @@
Code
-
-
-
-
-
-
- Data
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
@@ -203,185 +113,68 @@
Abstract
- We present the first method capable of photorealistically reconstructing a non-rigidly - deforming scene using photos/videos captured casually from mobile phones. -
-- Our approach augments neural radiance fields - (NeRF) by optimizing an - additional continuous volumetric deformation field that warps each observed point into a - canonical 5D NeRF. - We observe that these NeRF-like deformation fields are prone to local minima, and - propose a coarse-to-fine optimization method for coordinate-based models that allows for - more robust optimization. - By adapting principles from geometry processing and physical simulation to NeRF-like - models, we propose an elastic regularization of the deformation field that further - improves robustness. + Deep neural networks (DNNs) have become ubiquitous in machine learning, but their energy consumption remains problematically high. An effective strategy for reducing such consumption is supply-voltage reduction, but if done too aggressively, it can lead to accuracy degradation. This is due to random bit-flips in static random access memory (SRAM), where model parameters are stored.
- We show that Nerfies can turn casually captured selfie - photos/videos into deformable NeRF - models that allow for photorealistic renderings of the subject from arbitrary - viewpoints, which we dub "nerfies". We evaluate our method by collecting data - using a - rig with two mobile phones that take time-synchronized photos, yielding train/validation - images of the same pose at different viewpoints. We show that our method faithfully - reconstructs non-rigidly deforming scenes and reproduces unseen views with high - fidelity. + To address this challenge, we have developed NeuralFuse, a novel add-on module that handles the energy-accuracy tradeoff in low-voltage regimes by learning input transformations and using them to generate error-resistant data representations, thereby protecting DNN accuracy in both nominal and low-voltage scenarios. As well as being easy to implement, NeuralFuse can be readily applied to DNNs with limited access, such cloud-based APIs that are accessed remotely or non-configurable hardware. Our experimental results demonstrate that, at a 1% bit-error rate, NeuralFuse can reduce SRAM access energy by up to 24% while recovering accuracy by up to 57%. To the best of our knowledge, this is the first approach to addressing low-voltage-induced bit errors that requires no model retraining.
-
-
-
- Video
-
-
-
-
-
-
@@ -389,11 +182,13 @@
@@ -402,11 +197,10 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Matting
-
-
-
-
-
- - As a byproduct of our method, we can also solve the matting problem by ignoring - samples that fall outside of a bounding box during rendering. -
- -
-
+
-
+ 
+
+
Animation
- - -Interpolating states
-
-
+
-
+- We can also animate the scene by interpolating the deformation latent codes of two input - frames. Use the slider here to linearly interpolate between the left frame and the right - frame. +
Our Contributions
+
+
- Boosts DNN Accuracy Under Low Power + NeuralFuse improves the accuracy of deep neural networks (DNNs) operating in low-power environments with random bit errors, without needing to retrain the models.
-
- - - -
+
- 
-
-
- Start Frame
-
-
-
- Loading...
-
-
-
- 
-
+
- End Frame
-
+
- Protects DNN Accuracy Under Unstable Power + NeuralFuse improves the accuracy of deep neural networks (DNNs) operating in low-power environments with random bit errors, without needing to retrain the models. +
- - - -
Re-rendering the input video
-
-
-
+ - Using Nerfies, you can re-render a video from a novel - viewpoint such as a stabilized camera by playing back the training deformations. +
+
- Adapts to Limited-Access Settings + NeuralFuse supports deployment in scenarios with limited access to model details, using flexible training methods to adapt effectively across diverse DNN architectures.
-
+
-
+
-
-
Reduces Energy Use with Proven Performance + NeuralFuse recovers up to 57% of lost accuracy and reduces memory access energy by up to 24%, tested across diverse models (ResNet18, ResNet50, VGG11, VGG16, and VGG19) and datasets (CIFAR-10, CIFAR-100, GTSRB, and ImageNet-10). +
-
-
- Related Links
- +NeuralFuse Performance
+Energy/Accuracy Tradeoff
- There's a lot of excellent work that was introduced around the same time as ours. -
-- Progressive Encoding for Neural Optimization introduces an idea similar to our windowed position encoding for coarse-to-fine optimization. -
-- D-NeRF and NR-NeRF - both use deformation fields to model non-rigid scenes. -
-- Some works model videos with a NeRF by directly modulating the density, such as Video-NeRF, NSFF, and DyNeRF -
-- There are probably many more by the time you are reading this. Check out Frank Dellart's survey on recent NeRF papers, and Yen-Chen Lin's curated list of NeRF papers. + On the same base model (ResNet18), we illustrate the energy/accuracy tradeoff of six NeuralFuse implementations. +The x-axis represents the percentage reduction in dynamic-memory access energy at low-voltage settings (base model protected by NeuralFuse), as compared to the bit-error-free (nominal) voltage. The y-axis represents the perturbed accuracy (evaluated at low voltage) with a 1% bit-error rate.
+ BibTeX
-@article{park2021nerfies,
- author = {Park, Keunhong and Sinha, Utkarsh and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Seitz, Steven M. and Martin-Brualla, Ricardo},
- title = {Nerfies: Deformable Neural Radiance Fields},
- journal = {ICCV},
- year = {2021},
+ @inproceedings{sun2024neuralfuse,
+ title={{NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes}},
+ author={Hao-Lun Sun and Lei Hsiung and Nandhini Chandramoorthy and Pin-Yu Chen and Tsung-Yi Ho},
+ booktitle = {Advances in Neural Information Processing Systems},
+ publisher = {Curran Associates, Inc.},
+ volume = {37},
+ year = {2024}
}