Readme update - file streaming

[jendakol]

Howdy,
I'd like to offer you an update of your readme.

First of all, thanks for the great job and that includes the readme. In spite of that, I believe I've found a way how to improve it a little bit :-)

TLDR: I want to avoid someone will spend hours solving such a stupid issue.

Full story:
I played with ESP8266 and tried to serve npm/webpack built JS app from it. The output was two big files (cca 70kB JS and 30kB CSS).
When I tried to use server.serveStatic("/", SPIFFS, "/w").setDefaultFile("index.html"), the files were served very slowly and sometimes not even completely - developer console in the browser has reported "Failed - content-length mismatch", like if the ESP just cut the connection before spilling the whole file. I believe it was because ESP8266 has too low RAM available and it was trying to buffer too big part of those files.
I spent several hours with splitting those files into multiple ones which made things slightly better - usually, they were served just OK, even though very slowly. I tried to use request->beginResponse with callback instead (that seemed to me like the best/most intuitive/most efficient solution) however I've been fighting "exception 9".
After some time, I have solved the issue - it's not exactly rocket science but I hate idea someone will repeat my mistakes and waste time on it when it can be written just fine in a very pleasant time. The application now works very well and it's not even necessary to split those resources files.

With this PR, I'm adding two new sections to your readme which should help someone facing the same issues and possibly repeating my steps.

I'd like to declare I'm not a C/C++ programmer so if the problem with lambda can be solved somehow better, I'd love to hear it and change the PR accordingly.

[simkard69]

Hello and thanks for your input.

I'm actually trying to send chunkedresponse from SD files that are heavy (in terms of ESP32 : 100kB).
What do you mean by "calling the "File file" from a lambda ?

May I ask for your help/more details on that one ?

Thanks

[jendakol]

@simkard69 Hi, where did you read that phrase? Did you see the content of this PR?

[simkard69]

Hey, sorry it wasn't the latest version ... my bad.

Anyway, I'm trying to serve files from SD and things get very slow.
I'm using the SD library and SD-card is connected using SPI pins.

When I request a file of 161kB, it starts to download very slowly and ESP32 ends up crashing with this :

[HTTP]	BYTES [5637]	TOTAL [0]	SIZE [163873]	BUFFER_MAX_LENGHT [5637]
[HTTP]	BYTES [2872]	TOTAL [5637]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [8509]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [11381]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [14253]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [17125]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [19997]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
[HTTP]	BYTES [2872]	TOTAL [22869]	SIZE [163873]	BUFFER_MAX_LENGHT [2872]
E (310615) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (310615) task_wdt:  - async_tcp (CPU 0/1)
E (310615) task_wdt: Tasks currently running:
E (310615) task_wdt: CPU 0: IDLE
E (310615) task_wdt: CPU 1: loopTask
E (310615) task_wdt: Aborting.

Here is my code :

server.on("/css/bootstrap.min.css", HTTP_GET, [](AsyncWebServerRequest *request){
      //request->send(SD, "/webui/css/bootstrap.min.css", "text/css");
      // <<< INITIALIZE FILE DESCRIPTOR ON SD FILE >>> File f = SD.open(requested_file, FILE_READ);
      const File CSSfile = SD.open("/webui/css/bootstrap.min.css", FILE_READ);
      //const CSScontentType = "text/css";
      //AsyncWebServerResponse *response = request->beginChunkedResponse(
      AsyncWebServerResponse *response = request->beginResponse(
        "text/css",
        CSSfile.size(),
        [CSSfile](uint8_t *buffer, size_t maxLen, size_t total) -> size_t {
          File LOCALfile = CSSfile;                                   // Local copy of file pointer
          int bytes = LOCALfile.read(buffer, maxLen);                 // Read data from file until buffer
          if (bytes + total == LOCALfile.size()){                     // Close file at the end
            LOCALfile.close();
          }
          Serial.printf("[HTTP]\tBYTES [%d]\tTOTAL [%d]\tSIZE [%d]\tBUFFER_MAX_LENGHT [%d]\r\n", bytes, total, LOCALfile.size(), maxLen);
          return max(0, bytes);                                       // Return 0 even when no bytes were loaded
        }
      );
      request->send(response);
    }
  );

Is there something I do wrong from your perspective ?

[jendakol]

Disclaimer: I am far from being a C++ expert. TBH I hate it, it's an ugly language where you shoot yourself into the foot and find it out a week later. I code in it just because of the ESP programming.

I don't exactly understand why you use the LOCALfile variable but I don't see why it should be a problem either.
I don't see anything wrong with your code, you can check my working code. However, if you're struggling with the WatchDog and you're running in ESP32 tasks, it might be a good idea to relieve the WD by calling sth like

    taskYIELD();
    vTaskDelay(pdMS_TO_TICKS(1));

By that, you give up your CPU time, WD marks the thread is not blocked and you can happily continue with whatever you're doing. I call this basically in every loop i do on ESP32.

[simkard69]

By any chance this library can send data like on a chunked style without interrupting the rest of code (and that the loop() can continue to do its tasks) ?
I'm pretty sure there is a way to do that, but as I'm no C++ expert too, and that the repository's owner doesn't seem to act on it anymore ... captain, we have a problem here XD

[jendakol]

There could be a way, I don't know it... so far I always won the battle but yeah, it wasn't easy. I wish you strong will, faith and luck :-)
And yes, this library is abandoned. This PR is ready to be merged but it hangs here for 2 years already. Unfortunately, I still didn't find any better library than this one.

[simkard69]

Ok I think I found out my problem after all ... I'm quite ashamed by the root cause I finally found.
I'm using SD card with SPI connection (not SD_MMC which uses H/W 1bit or 4bit modes) and I created a benchmark test which ended up with pretty bad bandwidth.
It almost reads/writes to the SD card @ maximum 100 kilo-Bytes per MINUTE !!!

So now I think I have to work on that before trying to do anything else.
I will report back here as soon as I find a way to have better bandwidth.
For now, the best track I have is probably changing this :
SD.begin(<CS_PIN_OF_SD_CARD>);
to something like this :
SD.begin(<CS_PIN_OF_SD_CARD>, <HW_SPI_PORT>, <SPI_CLOCK>);
Where we can have those settings :

• <CS_PIN_OF_SD_CARD> : Whatever PIN you have the SD connected on
• <HW_SPI_PORT> : [HSPI] or [VSPI] for ESP32
• <SPI_CLOCK> : Hz number up until 80000000 (ESP32 can play SPI @ 80 Mhz easily)

[simkard69]

Ok so it seems that I'm running into some weir problem.
Using file.write(VALUE) and file.read(), I get :
~1250bps
Using file.write(BUFFER, LENGTH) I get : 5Kbps (results are quite the same for file.read(BUFFER, LENGTH)
2 MB written using 512 byte buffer for 418080 ms @ 5.000000 KBps

At first I thought the culprit was the 2GB SD card I put in which could be a slow model, so I switched to a U3, but it goes exactly the same ... where is the bottleneck ... ?

From what I can see here, I definitely have a problem somewhere : https://www.instructables.com/Select-SD-Interface-for-ESP32/

[jendakol]

You can bet SD is not the bottleneck, I just can't imagine that slow card 😄 As you may have noticed, I used SPIFFS in internal memory instead of SD card and it was slow as hell as well.Reading alone might be fine but streaming that into HTTP connection...
I think it's the ESP itself. While the CPU power is not that bad, it's just terrible for processing a bigger amount of data (and the bigger here means hundreds of kB). Or did you run the same benchmark as the Instructables did? 🤔 I'll try it do run it by myself too. Maybe later today...

[simkard69]

Switched the "core debug level" to Debug made me saw some interesting things down here :

...
[W][sd_diskio.cpp:174] sdCommand(): no token received
[W][sd_diskio.cpp:174] sdCommand(): no token received
[W][sd_diskio.cpp:174] sdCommand(): no token received
[W][sd_diskio.cpp:174] sdCommand(): no token received
...

I had some doubts about the micro-SD card slot so I totally removed all the resistors and capacitors, but it still acts the same.
I'm really having doubts on everything, but as far as I can see, nothing seems to be hardware related.

Ah yes and last thing regarding SPIFFS : it works very fine with my code.
file of 160 kilo-Bytes loads very fast in Chrome as you can see (164 kilo-Bytes in only 376mS including connection time etc etc ...) :

Here is my code for the "server.on......." part regarding SPIFFS and SD.
And YES, I managed to get "beginChunkedResponse" running fine. :)

// ========== SPIFFS ==========
  server.on("/spiffs/bootstrap.min.css", HTTP_GET, [](AsyncWebServerRequest *request){
      const File SPIFFSfile = SPIFFS.open("/bootstrap.min.css", FILE_READ);
      if (SPIFFSfile){
        Serial.printf("[HTTP]\tSPIFFS File exists [%d]\r\n", SPIFFSfile);
      } else {
        Serial.printf("[HTTP]\tSPIFFS File DOESN'T exists [%d] <<< ERROR !!!\r\n", SPIFFSfile);
      }
      AsyncWebServerResponse *response = request->beginChunkedResponse("text/css", [SPIFFSfile](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {
          File SPIFFSLambdaFile = SPIFFSfile;                                   // Local copy of file pointer
          Serial.printf("[HTTP]\t[%d]\tINDEX [%d]\tBUFFER_MAX_LENGHT [%d]\r\n", index, SPIFFSLambdaFile.size(), maxLen);
          return SPIFFSLambdaFile.read(buffer, maxLen);
        }
      );
      request->send(response);
    }
  );
  // ========== SD ==========
  server.on("/sd/bootstrap.min.css", HTTP_GET, [](AsyncWebServerRequest *request){
      const File SDfile = SD.open("/webui/css/bootstrap.min.css", FILE_READ);
      if (SDfile){
        Serial.printf("[HTTP]\tSD File exists [%d]\r\n", SDfile);
      } else {
        Serial.printf("[HTTP]\tSD File DOESN'T exists [%d] <<< ERROR !!!\r\n", SDfile);
      }
      AsyncWebServerResponse *response = request->beginChunkedResponse("text/css", [SDfile](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {
          File SDLambdaFile = SDfile;                                   // Local copy of file pointer
          Serial.printf("[HTTP]\t[%d]\tINDEX [%d]\tBUFFER_MAX_LENGHT [%d]\r\n", index, SDLambdaFile.size(), maxLen);
          return SDLambdaFile.read(buffer, maxLen);
        }
      );
      request->send(response);
    }
  );

[simkard69]

Ok so after a little bit of research and posts it seems that v2.0.2 of arduino-esp32 created the problem I'm facing ... lovelyyy !
Version 1.0.6 shouldn't contain the infamous bug.

Here is a post I've created with some informations regarding that : espressif/arduino-esp32#6338

[simkard69]

Well, applied bugfix on top of v2.0.2 of arduino-esp32 aaaaaaaaaaaaaand ...

Magically, everything works as fast as it should !

I even ran again my benchmarks, here are my results using simple file.read() and file.write(value) :

• WRITE : 100kB took 1351 mS (approx. 76 kilo-Bytes per second)
• READ : 100kB took 1163 mS (approx. 88 kilo-Bytes per second)
• ... all well better than 1250 bytes per second XDDD

Everything runs fine and smooth now, so happy that the error wasn't my culprit XDDDD

For those who are running into the same problem, in v2.0.3 of arduino-esp32 this is corrected.
Anyway, for those like me who needs to bypass this problem, you can follow this : https://github.com/espressif/arduino-esp32/pull/6103/files

Cheers to everyone !

[jendakol]

Nice, glad to hear you solved it!

[guestisp]

Still not working here, even using the code from the linked project, by using the 404 callback
Everything is very slow (almost unusable) and still getting the content length mismatch errors

[Belleson]

#response_container_BBPPID{font-family: Tahoma, Geneva, sans-serif; font-size:11pt; color: #1F497D;} Thanks,Brian LoweBelleson LLC From: ***@***.***: April 28, 2022 5:34 AMTo: ***@***.***: ***@***.***: ***@***.***: Re: [me-no-dev/ESPAsyncWebServer] Readme update - file streaming (#770) Still not working here, even using the code from the linked project, by using the 404 callback Everything is very slow (almost unusable) and still getting the content length mismatch errors —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[davepl]

Who and what and when is the SPIFFS file closed? In my own code, I do it when I hit EOF, but not sure this code even does it. Or is it handled automatically somehow?