The UI is somewhat tricky...but yes. Admin => Misc => Reset Container Dashboard. Put in the Container ID and Type, and select Load Widgets. The index numbers that show in each box represent column position (left to right) (i.e. 1 or 0 is the left most column) .... same goes for Widget in the column...1 or 0 is the top most widget. Use you knowledge of the widget's location to remove the affected widget (click on the widget ID hyperlink). When done, click on "Publish Draft Widgets".
This is a very tested feature in the tool, but never got around to making the UI remotely user-friendly. Please let me know if this doesn't help.
P.S. I made this feature for this exact use-case. =)