Project

General

Profile

USB serial in Forth

Added by ekoeppen about 1 year ago

Great you got this working! I cleaned up my code a bit more, and got the lower parts running on an STM32F072 (that chip looks very nice in general, crystal-less USB and built-in bootloaders - and it has a slightly more sensible access to the USB PMA). I'll hopefully get it soon posted.

I solved the USB output stalling issue in another project by looking at the CDC messages, some of those can be used to detect the connection properly. You could use e.g. the set line state message, that should only come once you've gotten a connection on the other end.

The input overrun should be fixable by setting the out (RX) endpoint status to NAK if the ring buffer is starting to be full.


Replies (16)

RE: USB serial in Forth - Added by jcw 9 months ago

Made some good progress with the USB driver again. But it's far from finished:

https://github.com/jeelabs/embello/tree/master/explore/1608-forth/suf

To get this to work, you need either a HyTiny or an Olimexino-STM32 (and other Leaflabs Maple-like boards), or a "generic" STM32F103 board - there are different HEX firmware images, because each board has a different way of pulling down the D+ line to trigger USB enumeration:

  • usb-generic.hex pulls PA12 down, i.e. the D+ line itself
  • usb-hytiny.hex pulls PA0 up to pull D+ down (via an on-board transistor)
  • usb-olimexino.hex pulls PC12 up to pull D+ down (via an on-board transistor)

The Mecrisp core takes 20 KB and the USB console driver takes just over 5 KB. The eraseflash word has been redefined to never erase anything below 26 KB ($6800) - so unless init is re-defined improperly, this code should always reset into a usable interactive mode over USB.

There are still a whole bunch of gotcha's with this code:

  • after being powered up, the board needs an extra reset to actually appear as USB device (no idea why... timing?)
  • hitting reset always disconnects and re-enumerates the USB connection, i.e. you have to re-open the serial connection after pressing reset or calling the reset word - this is similar to the Arduino Leonardo, and inherent in the fact that the same µC is used for both USB and running application code
  • data sent to the µC needs to be throttled slightly to avoid losing USB packets - I haven't figured out how to prevent this yet, but note that typing manually using a terminal emulator is so slow that it won't run into this issue
  • the Folie command-line utility has been tweaked to insert 2 ms delays when lines longer than 60 characters are being sent - since it already does all sorts of throttling to avoid over-running the Forth interpreter input loop, this seemed like an acceptable workaround for now
  • when there is no serial connection, i.e. the plugged into USB but no serial terminal session is active, console output from Forth will block and stop whatever is running (but interrupt handlers will continue to run)
  • this blocking behaviour is not a show-stopper during development, but makes the board harder to use with only USB power - there needs to be a way to detect connection loss and discard data in this case (just like serial cables just drop all output when nothing is connected)

Some more notes w.r.t. development and debugging:

  • there is a safety escape mechanism to disable the USB driver and keep the serial console active: if you hit any key (on serial!) within the first second after resetting the board, the usb-io word won't be called to switch to USB
  • with Folie, it's very easy to re-install a (modified) driver via the serial port: enter include f-hytiny.fs (or whatever board you're using), and it'll erase everything above 20 KB and re-upload/-compile the driver from scratch - at the end, a huge hex dump is printed, which you can ignore - then just press reset and the new code will start

Having said that, the USB serial console does work nicely: a new USB device shows up, you can connect to it, and the baudrate is ignored, since it runs at the maximum speed the USB link will support (up to 12 Mbps). Try typing words to see just how much faster the link is than the default serial 115,200 baud hookup.

I hope this has enough information for some early adopters / USB gurus / brave souls / kind contributors to start playing with this setup. There are lots of STM32F103-based boards which could be turned into interactively-programmable Forth engines once we squash these remaining bugs and solidify this USB console driver for Mecrisp Forth.

Enjoy,
-jcw

RE: USB serial in Forth - Added by jcw 9 months ago

after being powered up, the board needs an extra reset to actually appear as USB device

Looks like that's not the case for generic boards - those work right away when plugged in.
Maybe there's an issue with the inverted logic starting out wrong in the other cases.

I've now got a bunch of different F103 boards working on USB, including these from eBay.

RE: USB serial in Forth - Added by jcw 9 months ago

Well ... on Mac it works, but on Ubuntu Linux, it won't:

[  629.107128] usb 1-1.2: new full-speed USB device number 6 using dwc_otg
[  629.184691] usb 1-1.2: device descriptor read/64, error -32
[  629.374584] usb 1-1.2: device descriptor read/64, error -32
[  629.564642] usb 1-1.2: new full-speed USB device number 7 using dwc_otg
[  629.644536] usb 1-1.2: device descriptor read/64, error -32
[  629.835625] usb 1-1.2: device descriptor read/64, error -32
[  630.024520] usb 1-1.2: new full-speed USB device number 8 using dwc_otg
[  630.446468] usb 1-1.2: device not accepting address 8, error -32
[  630.527267] usb 1-1.2: new full-speed USB device number 9 using dwc_otg
[  630.945740] usb 1-1.2: device not accepting address 9, error -32
[  630.949353] hub 1-1:1.0: unable to enumerate USB device on port 2

So don't expect too much at this stage. I'd love to get some help with this, but it sure ain't ready for prime time...

RE: USB serial in Forth - Added by JohnO 9 months ago

My Mac isn't able to enumerate although the generic STM board works on Win7.

USBF:    24381.569    The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 1 of Hub at 0x14000000)
USBF:    24381.870    The IOUSBFamily was not able to enumerate a device.
         0 [Level 5] [com.apple.message.domain com.apple.commssw.cdc.device] [com.apple.message.signature AppleUSBCDCACMData] [com.apple.message.signature2 0x483] [com.apple.message.signature3 0x5740]
AppleUSBCDCACMData: Version number - 4.3.2b1, Input buffers 8, Output buffers 16
AppleUSBCDC: Version number - 4.3.2b1

(edited to use "```" iso "~~~" around code blocks -jcw)

RE: USB serial in Forth - Added by jcw 9 months ago

Odd - could you try the latest build from minutes ago, which addresses a startup timing issue?
Also, what happens if you connect through a powered or unpowered USB 2 hub?

Meanwhile, the last timing fix seems to have solved a few cases on Mac and Linux for me.
But enumeration/startup is still not reliable, even with the same board.

One of the odd behaviours I see is that a board with some other circuitry attached works fine, whereas the same type of board without doesn't. There may be more timing issues, dependencies on how power ramps up, or just plain sensitivity to different tolerances in each chip or circuit.

RE: USB serial in Forth - Added by JohnO 9 months ago

Latest build still works fine on Win7 but Mac shows:

USBF:    3155.862    The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 1 of Hub at 0x14000000)
USBF:    3156.163    The IOUSBFamily was not able to enumerate a device.
USBF:    3157.178    The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 1 of Hub at 0x14000000)
USBF:    3157.479    The IOUSBFamily was not able to enumerate a device.
USBF:    3158.493    The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 1 of Hub at 0x14000000)
USBF:    3158.794    The IOUSBFamily was not able to enumerate a device.
USBF:    3159.810    The IOUSBFamily is having trouble enumerating a USB device that has been plugged in.  It will keep retrying.  (Port 1 of Hub at 0x14000000)
USBF:    3160.332    The IOUSBFamily was not able to enumerate a device.

I don't have a hub to hand, will scout around for one.

RE: USB serial in Forth - Added by JohnO 9 months ago

I am still running Yosemite.

RE: USB serial in Forth - Added by jcw 9 months ago

Thx - 10.10 and 10.11 should be ok, 10.12 (El Capitan) probably won't make a difference.

RE: USB serial in Forth - Added by JohnO 9 months ago

With a very suspect USB hub in place I have:
Bus 026 Device 006: ID 0483:5740 STMicroelectronics Forth Serial Port Serial: 31262E15

RE: USB serial in Forth - Added by jcw 9 months ago

Good - looks like you should be able to connect to /dev/cu.usbmodem31262E11 now.

RE: USB serial in Forth - Added by JohnO 9 months ago

Excellent, that works fine.

RE: USB serial in Forth - Added by jcw 9 months ago

Looks like the latest build now also works more reliably on Linux:

[   36.564885] usb 1-1.2: new full-speed USB device number 6 using dwc_otg
[   36.655641] usb 1-1.2: device descriptor read/64, error -32
[   36.845886] usb 1-1.2: device descriptor read/64, error -32
[   37.035844] usb 1-1.2: new full-speed USB device number 7 using dwc_otg
[   37.179609] cdc_acm 1-1.2:1.0: This device cannot do calls on its own. It is not a modem.
[   37.185205] cdc_acm 1-1.2:1.0: ttyACM0: USB ACM device
[   37.188807] usbcore: registered new interface driver cdc_acm
[   37.193639] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters

Still some error messages, but it enumerates properly every time I try it.

In summary: the latest code works on Mac, Windows, and Linux (and does not need a driver!).
At least for some combinations of board type, USB 2 vs 3, OS version, and moon phase.

Onwards!

RE: USB serial in Forth - Added by ekoeppen 9 months ago

In order to increase throughput and handle the overrun issues in a more direct manner, I've played around with an alternative endpoint implementation (in C++ so far only). Instead of ringbuffers, I'm passing application level buffers directly to the endpoint. This reduces the times the data needs to be copied around to one, from the application buffer to the packet memory area.

On the transmitting side, it requires the application to wait for the current buffer to be sent completely. The endpoint maintains a pointer to the remaining buffered data and moves that forward on each IN packet. It also keeps track of the number of remaining bytes, and once that's zero, the application can feed in another buffer.

The receiving side in the endpoint fills an application provided buffer. In the OUT handler, it first sets the endpoint state to NAK, and only if there is still space in the buffer, it sets it back to valid. This achieves flow control, because the host will simply retry NAK'ed packets.

This is all single buffered, but the STM32F1 USB block can also set up for double buffering. That could be interesting to try as well :)

RE: USB serial in Forth - Added by jcw 9 months ago

Thanks Eckhart, but I think you're talking about speed optimisations.
I'm not quite that far - just trying to get the driver robust.

Right now, everything I throw at it works, under the following constraints:

  • no back-to-back packets from host to µC - adding 2ms delays after every 60 bytes or so makes it work properly
  • keep a session open on the host when the µC is sending output - else it'll block

Plug/unplug on macOS and Linux works (but w/ very limited use on Linux so far), and so does Win7, apparently.

I see very occasional failures to show up (i.e. enumerate) when plugged in, but a retry always fixes it.

I've extended the Folie utility to insert those 2ms delays, and to reconnect when the connection is dropped (which happens on a s/w or h/w reset of the µC, among other cases). The result actually works quite smoothly - despite these quirks.

While the idea to drop extra buffers sounds good, I'm not sure it affects the above issues. And in a way, a ring buffer is really just another variation on double-buffering.

Output works really well, by the way - and is blindingly fast compared to the 115,200 baud rate of std Mecrisp serial console I/O.

The no-back-to-back bug puzzles me. I don't understand how the driver loses data, since as far as I can tell, the hardware will send NAKs as long as it can't receive another buffer. Maybe I'm telling it that more can be accepted too soon? - for anyone who wants to have a look, here are the lines involved in pickup up a data packet:

https://github.com/jeelabs/embello/blob/master/explore/1608-forth/suf/usb.fs#L262-L269

But be warned. This is terrifyingly tricky code - there's no other way to put it. My main reference is the "RM0008" reference manual for the STM32F103 chip, and its chapter on USB is truly, ehm... weird?

RE: USB serial in Forth - Added by ekoeppen 9 months ago

The move to application level buffers was for speed, but it did help with the NAK as well :)

When looking at ep-out, I can see that you set the endpoint to VALID via ep-reset-rx#, but I can't seem to find where you set the state to NAK. I was thinking you could add a check in ep-out to set the state to NAK as soon as the ringbuffer has less than 50% capacity, and add some logic to set the state back to VALID in usb-key (i.e. when you remove bytes from the ringbuffer).

I haven't pushed my code yet, but this is what I'm doing in the IRQ handler for reading the data out:

        static bool receive_rx_data(void) {
                bool r = false;
                EP::set_rx_status(NAK); // endpoint is not accepting new data for now
                if (rx_current < rx_count && rx_data != NULL) { // rx_current points to last received byte in application level buffer, rx_count is the available space, and rx_data is the buffer
                        uint32_t count;
                        uint16_t *data = EP::rx_data_addr();
                        ep_rx_pending = EP::get_rx_count(); // read the number of available bytes in the EP buffer
                        count = MIN(ep_rx_pending, rx_count - rx_current);
                        ep_to_rx_data(count, data); // copy bytes into the application buffer, increasing rx_current along the way
                        if (rx_current < rx_count) {
                                EP::reset_rx_count();
                                EP::set_rx_status(VALID); // we have space, set the status back to VALID
                        } else {
                                r = true;
                        }
                } else {
                        EP::reset_rx_count();
                }
                return r;
          }

RE: USB serial in Forth - Added by jcw 9 months ago

Ooh... I sense light at the end of this tunnel. Will investigate - MANY thanks!

    (1-16/16)