usb: musb: only remove host/udc if it has been added

musb_shutdown() removes always USB host and device.
musb_init_controller() adds host and device depending on port_mode. If
port mode is set to HOST then the removal of UDC leads only to:
|(NULL device *): gadget not registered.
and nothing else happens. If port mode is set to DEVICE and we remove
the host then we oops in usb_remove_hcd().
This patch ensures that we only remove host in OTG/host mode and device
only in OTG/device mode to avoid any trouble.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Felipe Balbi <balbi@ti.com>
diff --git a/drivers/usb/musb/musb_host.c b/drivers/usb/musb/musb_host.c
index a9695f5..9a2b8c8 100644
--- a/drivers/usb/musb/musb_host.c
+++ b/drivers/usb/musb/musb_host.c
@@ -2628,6 +2628,8 @@
 
 void musb_host_cleanup(struct musb *musb)
 {
+	if (musb->port_mode == MUSB_PORT_MODE_GADGET)
+		return;
 	usb_remove_hcd(musb->hcd);
 	musb->hcd = NULL;
 }