I solved the issue by setting the navigation of the buttons in the grid manually instead of letting it escape to the widget next to it.