We have come across a very bizarre issue whereby front-end data throughput to a Netapp Altavault appliance randomly reduces until the services have restarted.
It took about a month of troubleshooting to pinpoint the issue with Netapp Support, which finally came down to disabling TOE (TCP Offload Engine) settings on the Altavault.
The Netapp Altavault is connected to two Cisco Nexus 5548 switches and while TOE is supposed to help increase speed, in this case, it does quite the opposite.
The behavior of the Altavault front-end throughput looks like the following:
The reduced rate will continue until the Altavault services are restarted.
If you are experiencing similar behaviour you will need to log a support case with Netapp and ask them to assist you in disabling the TOE settings for your data interfaces. Data interfaces include stand-alone and interfaces that are members of a vif (port-channel).
Disabling TOE settings on Altavault Data Interfaces
SSH to the Altavault device and enter into enable mode
# cli challenge generate (Generate a challenge code for Netapp)
# cli challenge response (Enter in Netapp challenge response code)
@> _shell (enter the Altavault shell)
@> ethtool -k e0a (check tso, gso, gro, lro settings on interface e0a)
@> ethtool -K e0a tso off gso off gro off lro off (disable's offload settings for e0a)
@> mount (check if /dev/loop- on / is ro)
@> mount -o remount,rw / (make / rw)
@> mount | grep loop0 (check / is now rw)
@> vi /etc/rc.local (edit rc.local and enter in the 2 commands below which disables TOE on e0a and e0c)
/usr/sbin/ethtool -K e0a tso off gso off gro off lro off /usr/sbin/ethtool -K e0c tso off gso off gro off lro off
add the 2 lines above below – touch /var/lock/subsys/local but above /sbin/kernel_crashdump.sh
@> mount -o remount,ro / (make / ro)
No restart is required
After disabling TOE on the data interfaces you will instantaneously see the front-end speed return to normal speeds
I’d be interested to know if anyone else is experiencing this issue on switches other than Cisco, please leave a comment.
The post Netapp Altavault Data Throughput Reduces Randomly appeared first on SYSADMINTUTORIALS IT TECHNOLOGY BLOG.