ESP32 Based Web Interface & Data Logger

[Pete9008]

The esp codes not fussy about what comes back, whatever it receives gets logged to the sd card. As long as you have a way of decoding the files from the sd card the stm can send just about anything you like when in logging mode.

[johu]

Sure, you'd just need to send "binstream -1 il1,il2" once

[Pete9008]

Following from this thread viewtopic.php?t=3520 had a thought on how to make the logging a bit more flexible.

Essentially the idea is to add a second logging mode that uses the standard 'stream' command instead of binary logging. This would work in the same way as the binary logging in that when there were no WiFi connections the ESP would send a stream command (the actual command would be read from a text file in the SPIFFS file system (so that it can be updated/changed via the existing web interface upload functionality). Any data received back would then be written to the SD card. The stream command results in a comma delimited ascii data stream which means the data on the card could be directly imported into any standard graphing or analysis software.

Would probably be worth adding a time stamp to the front of each line. Would also be worth adding the stream command used at the top of each SD card file to aid analysis.

This should work with completely standard OI firmware. The only possible change needed would be to increase the number of parameters that the stream command can handle from the current value of 10 to say 20 to increase the logging capability.

Not sure when I'll get to actually doing this though.

[johu]

I've started coding the CAN backend of ESP32 web interface but ran into some roadblocks: the CAN driver behaves weirdly when SDOs are requested in a for-loop. Instead of sending out 0x601#40 00 50 01 00 00 00 00 for example, it sends out and all zero message 0x601#0 0 0 0 0 0 0 0 0

If I do one request per call of the loop() it works as intended, I can even download the full json then. I use miwagner/ESP32CAN@^0.0.1
Now I'm not sure which way to go now, bug hunt the CAN issue or try to convert the data paths to WebSocket.

[Pete9008]

Might be worth a look at the CAN library gvret uses on the ESP32 https://github.com/collin80/ESP32RET , that's the one I hope to use.

Sounds like yours isn't interrupt based?

[arber333]

Might be worth a look at the CAN library gvret uses on the ESP32 https://github.com/collin80/ESP32RET , that's the one I hope to use.

Sounds like yours isn't interrupt based?

I am already using Collins code to run Outlander charger, DCDC, AC compressor, Heater and Tesla DCDC with ESP32 board.
I recommend using interrupts as this allows to use CAN telegrams out of sync.
I use timer interrupt to time telegrams to 100ms and cyclic central telegram 0x285 at 10ms.
I also use watchdog at 2s interval to verify chip behaves...
Code is here... https://github.com/arber333/ESP32-VCU/b ... _code2.ino

[uhi22]

There is a longer discussion of ESP32 CAN controller issues here: https://github.com/espressif/esp-idf/issues/4276
Not sure whether this is related to the topic here, and also have no clue whether there is a stable solution in the end. My understanding at the moment is: The CAN works fine, as long as there is nearly no bus load, and with increasing bus load, the troubles are starting, because the hardware buffer handling is not correctly designed.

[johu]

That could be what I'm seeing. I'll switch to Collins library now https://github.com/collin80/esp32_can and will handle messages in an interrupt, after handling send out the next request.

[johu]

This can interface is terrible! I've tried Collins driver but didn't manage to send any message with it.
Now I'm using the espressif twai api but even that is not working stable. I have no idea what it is that makes it work stable for one run and then at the next change it fails with another mode again.

Current source code attached.

Here's the traffic it creates:

Code: Select all

ID	Extended	Bus	LEN	D1	D2	D3	D4	D5	D6	D7	D8
601	false		1	8	40	0	50	0	0	0	0	0
581	false		1	8	43	0	50	0	56	FF	6D	6
601	false		1	8	40	0	50	1	0	0	0	0
581	false		1	8	43	0	50	1	51	83	57	49
601	false		1	8	43	0	50	1	51	83	57	49

So the last 601 message instead of requesting (0x40 in byte 0) is just a copy of the last received message. wtf...

UPDATE: I think I found it! The twai_message_t tx_frame mustn't reside on the stack, it must remain in memory until the frame is sent or enqueued or something. Now it works.

[johu]

Finally managed to implement the "json", "set" and "stream" command via CAN. Now trying to decide whether I should mix it in with the existing code or start with a fresh code base?
As it stands the UART comms is turned off. A unified web interface would let you choose perhaps between UART0, 1, 2 or CAN (nodeid 1-63)

UPDATE: https://github.com/jsphuebner/esp32-web-interface
I left the uart code in there but removed the binary logging as that won't work over CAN or would cover a different use case.

[johu]

Added update via CAN also. It runs quite quickly.
https://github.com/jsphuebner/esp32-web ... 73fc33220b

EDIT: I should add, the update now runs autonomously once it has been started. So even if the wifi connection collapses the update will finish.

Now Save/Restore/Stop etc. commands to be added and also CAN mapping via CAN Needs a better implementation in canmap.cpp as well

[johu]

[Pete9008]

Not sure whether there is still any interest in the data logging side of this, if so then there is a bit of progress described over in my build thread (viewtopic.php?p=57268#p57268).

If there is any interest let me know and I'll look at porting the code back to the F103.

[Bigpie]

I am for sure. Picked up a Yaris inverter to try your auto tune, binary logging and resistor mod outside of the car

[Pete9008]

Great stuff are you going to use it with the Leaf or the Outlander front motor?

I'll look at porting it over to the auto-setup branch (or another branch off that branch).

First I'm going to have a go at exporting the data from the decoder app directly to PulseView's native file format (mentioned in the other thread it's not displaying the data right and my suspicion is that it's the PulseView csv importer that's the problem).

[Bigpie]

Eventually leaf motor in a car, but while it's in garage I'll run auto tune on leaf and outlander front motor.

[Pete9008]

The decoder app for the binary logs has now been tidied up and is on github: https://github.com/Pete9008/LoggingDecode

It's a Qt console application and now accepts command line arguments for input and output file names and options to select what processing is done. It can generate a JSON file containing the inverter parameters, two CSV files (one for high rate core control loop data and a lower rate one containing all spot parameters) and a PulseView native format file containing the core control loop data.

The default is to generate all output files and takes around 30sec to process 1min of input data. Skipping the motor data csv output file halves this (might be sensible to change the default to do this!).

It should be backward compatible with old logs but I haven't tried it. The spot data is a new feature and currently only works with my F405 build but will look at porting it back to the F103 code.

Note:
The old CSV PulseView import method has been found to corrupt the data so the new native format should always be used with PulseView.

The native PulseView format needs the data files to be zipped into a single archive. The program uses a command line call to do this so is only likely to work on Linux systems. To work on other platforms the command and arguments for the process call in the code will need to be changed to suit. Alternatively the .PulseViewZip folder could be manually zipped.

[Pete9008]

I am for sure. Picked up a Yaris inverter to try your auto tune, binary logging and resistor mod outside of the car

Any idea when you might be doing this? Might be interested in popping over when you do as it looks like the auto setup code may need a little work!

Edit - Also found an old Bluepill board in the cupboard so going to try and get some F103 logging hardware setup here.

[Bigpie]

Working on assembly of the boards at the moment. Just about there with the inverter board so hopefully by the weekend should have check the various power rails and verified PWM etc.

Need to get the esp32 BOM parts ordered next, but could just run web interface on a esp32 devboard mess from last time.

WFH so pretty much always around

[Pete9008]

Thanks, no rush at all so whenever you're ready to try it.

[Bigpie]

Got my test setup now. Need to debug my lack of resolver readings, next.

[Pete9008]

Is the resolver issue using the auto setup code or 5.27? If the former then it's sounding increasingly likely that I've messed something up in the code (Romale had a similar issue).

Got a logging setup here now using an old bluepill board but haven't got anything done on porting the code over yet.

[Bigpie]

I'll try a known working version later. I broke out the picoscope and it looked like the exciter was running. I didn't hear the pwm start either.
You're welcome to come mess around with my setup if you want.

EDIT. Put an old version on and still not working. So will have to go back over the Yaris board thread and debug.

[Pete9008]

You're welcome to come mess around with my setup if you want.

Thanks, that would be good.

Probably makes sense to get the logging code ported across first though (and I could do with getting my laptop sorted out too - needs the OS updating and all the dev tools putting on it).

[Proton]

This is good news because I really do not like the Wifi option. I do not like to leave a digital trail everywhere I go. I do not use wifi in my house on any devices.
I was thinking that after i am done setting up the motor, to open the motor and damage the wifi card so it will never transmit. for sure I will not leave it in there.
Hopefully I will be able to use the CAN solution.