Project

General

Profile

Resolved: Does ethercard browseurl callback duplicate callback with persistTcpConnection ?

Added by DaveOB over 4 years ago

Am now starting to understand the browseurl and callback now - but still can’t find an explanation for all the parameters in the callback function.

static void my_callback (byte status, word off, word len)

‘ status ’ - I can not find explanation for this - please help if you know what values can be passed here, and what they mean / represent.

IF I understand this correctly :

When browseurl is first called, it runs asynchronous, until ethernet data arrives, and that buffer is then passed to the callback function.

The first callback has the offset = 54 and len = 512.
If this is the first call, why is there an offset ? Should the data not start at position 1 in the buffer ?

Next, with the use of persistTcpConnection, the browseurl will continue to run until the end of the stream of data ( in 512k blocks ) from the web server arrives.

Does the callback wait to complete its code before the next callback is activated ?

What happens if the second browseurl modifies the buffer while the code in the callback is still busy working with it.

For example, in my callback I have :

static void my_callback (byte status, word off, word len) {
    Serial.println("my_callback initiated");
    Serial.print("Status : ");   Serial.println(status);
    Serial.print("off : ");         Serial.println(off);
    Serial.print("len : ");         Serial.println(len);
    Serial.println((const char*) Ethernet::buffer + off);

    Ethernet::buffer[off+len] = 0;  // after the offset and the length, add an end marker .. is this correct ?  Sample codes showed [off + 300]

    WriteSDfile(2,0,((char*) Ethernet::buffer + off)); // now I need to write ( append ) the data in the buffer to an SD card file ( file number 2, no erase, data to append to file )

    Ethernet::buffer[0] = 0;  // set the end market to the start of the buffer so next received block of data is at the start of the buffer

        if(len != 512){  // if this is the last block of data ( less than len 512 ) then echo the SD Card Data to the Serial Monitor
        Serial.println("......................................");
            Serial.println("Reading Contents of File 2");
        Serial.println("......................................");
            ReadSDfile(2);
        }   
}

Or, is the code in the callback run, and must complete all of its code in the function before the callback can accept new data in the buffer in the next loop of the script execution ?

If this is the case, can the ethernet module modify the data in the ( existing ) buffer while the first instance of the callback function is still trying to process the code.

Would it be better to copy the buffer to a second byte array at the start of the callback, clear the buffer ( Ethernet::buffer[0] = 0; ) and then work with the copy array ?

I would really appreciate any advice here.


Replies (4)

RE: Does ethercard browseurl callback duplicate callback with persistTcpConnection ? - Added by DaveOB over 4 years ago

SOLVED

Apart from still not understanding the status in the callback function, I think I now have this figured out.

( copy and paste from my post in the Arduino forum, just in case it helps someone else with similar problems )
http://forum.arduino.cc/index.php?topic=286822.msg2009648\#msg2009648

Am very glad to say I seem to have finally managed to solve this.

I am using an ethernet module ( ENC28J60 ) with the EtherCard library and SD Card both on SPI on a Mega 2560.

The CS pins are different for each, and pin 53 as an OUTPUT.
The project I want needs a web page ( output from a php script ) to be downloaded and the data saved to an SD card file.
The root of the problem is the async browseurl and the callback function.

From my understanding, the browseurl sends the web page request to the ethernet module, and then continues with it’s code.
During the main loop, the ethercard library then populates the ethernet buffer with the received data, and calls the callback function.

From my experiences, I can say it appears that this works great, UNLESS you actually do something with the data in the buffer in the callback function, like try and write it to an SD card file.

When this happens, the callback receives duplicates of the data packet. I can only assume that this is caused by the library ( or the ethernet module, not sure which ) not getting the message that the callback function has completed within a certain timeout period. It then sends a duplicate buffer.
I was able to confirm this theory by adding a delay (400ms) after writing the data to the SD Card file. This caused almost every single buffer to be duplicated.

My solution ( probably not the best, but works for me ) was to do the following :

I have 4 char arrays ( xBuf1, xBuf2, xBuf3, xBuf4 ) each 33 bytes.
When a buffer arrives ( 512 bytes ) make a MD2 hash of the data ( 33 bytes ).
Compare the 33 byte MD2 hash to the 4 xBuf arrays. ( 33 byte arrays use less memory to compare than would 4 x 700 byte arrays to store the complete buffer )
If there is a match, then this buffer was one of the last 4 buffers previously received, so do nothing with it. This will skip the SD Card File write, and the callback function completes faster, and the buffer is never repeated.
If there is no match, then delete the oldest MD2 from the xBuf arrays, and add the MD2 of the new data to the xBuf array.

I hope this helps someone in the future.

RE: Does ethercard browseurl callback duplicate callback with persistTcpConnection ? - Added by DaveOB over 4 years ago

Thanks JohnO.

Not knowing the ‘status’ values / meanings is all I need now, I think.

Anyhow, here is my code if it is of any use to anyone. It is not neat, but it is working for me.

There are a lot of Serial.print lines that I added when trying to identify the problem, but may help someone like me understand the code better.

#include 
#include 
#include 

int LoopCount = 0;
long LastMil = 0;
int IgnoreCount = 0;

static byte mymac[] = { 0x74,0x69,0x70,0x2D,0x30,0x31 }; // ethernet interface mac address, must be unique on the LAN
static byte static_ip[] = { 192,168,1,219 }; //local IP of the ethercard module
static byte static_gw[] = { 192,168,1,254 }; // router IP address
static byte static_dns[] = { 192,168,1,254 }; // dns via router

byte Ethernet::buffer[700];  // buffer for ethercard to pass received data to callback function
byte ebuf2[700];  // array used to make a copy of Etehrnet::buffer for use in the callback

byte chx[6];

char xBuf1[33];  // these arrays are used to store the MD2 hash of the last 4 buffers received by the callback function
char xBuf2[33];
char xBuf3[33];
char xBuf4[33];
char md2str[33]; // this will be the MD2 of the current buffer

/* 
MD2   -- this code is copied from :
https://github.com/scottmac/arduino/blob/master/md2/md2.pde

*/
typedef struct {
    unsigned char state[48];
    unsigned char checksum[16];
    unsigned char buffer[16];
    char in_buffer;
} MD2_CTX;

static const unsigned char MD2_S[256] = {
     41,  46,  67, 201, 162, 216, 124,   1,  61,  54,  84, 161, 236, 240,   6,  19,
     98, 167,   5, 243, 192, 199, 115, 140, 152, 147,  43, 217, 188,  76, 130, 202,
     30, 155,  87,  60, 253, 212, 224,  22, 103,  66, 111,  24, 138,  23, 229,  18,
    190,  78, 196, 214, 218, 158, 222,  73, 160, 251, 245, 142, 187,  47, 238, 122,
    169, 104, 121, 145,  21, 178,   7,  63, 148, 194,  16, 137,  11,  34,  95,  33,
    128, 127,  93, 154,  90, 144,  50,  39,  53,  62, 204, 231, 191, 247, 151,   3,
    255,  25,  48, 179,  72, 165, 181, 209, 215,  94, 146,  42, 172,  86, 170, 198,
     79, 184,  56, 210, 150, 164, 125, 182, 118, 252, 107, 226, 156, 116,   4, 241,
     69, 157, 112,  89, 100, 113, 135,  32, 134,  91, 207, 101, 230,  45, 168,   2,
     27,  96,  37, 173, 174, 176, 185, 246,  28,  70,  97, 105,  52,  64, 126,  15,
     85,  71, 163,  35, 221,  81, 175,  58, 195,  92, 249, 206, 186, 197, 234,  38,
     44,  83,  13, 110, 133,  40, 132,   9, 211, 223, 205, 244,  65, 129,  77,  82,
    106, 220,  55, 200, 108, 193, 171, 250,  36, 225, 123,   8,  12, 189, 177,  74,
    120, 136, 149, 139, 227,  99, 232, 109, 233, 203, 213, 254,  59,   0,  29,  57,
    242, 239, 183,  14, 102,  88, 208, 228, 166, 119, 114, 248, 235, 117,  75,  10,
     49,  68,  80, 180, 143, 237,  31,  26, 219, 153, 141,  51, 159,  17, 131,  20 };

void MD2Init(void *contextBuf){
    MD2_CTX *context = (MD2_CTX*)contextBuf;
    memset(context, 0, sizeof(MD2_CTX));
}

static void MD2_Transform(void *contextBuf, const unsigned char *block){
    MD2_CTX *context = (MD2_CTX*)contextBuf;
    unsigned char i,j,t = 0;
    for(i = 0; i < 16; i++) {
        context->state[16+i] = block[i];
        context->state[32+i] = (context->state[16+i] ^ context->state[i]);
    }
    for(i = 0; i < 18; i++) {
        for(j = 0; j < 48; j++) {
            t = context->state[j] = context->state[j] ^ MD2_S[t];
        }
        t += i;
    }
    /* Update checksum -- must be after transform to avoid fouling up last message block */
    t = context->checksum[15];
    for(i = 0; i < 16; i++) {
        t = context->checksum[i] ^= MD2_S[block[i] ^ t];
    }
}

void MD2Update(void *contextBuf, const unsigned char *buf, unsigned int len){
    MD2_CTX *context = (MD2_CTX*)contextBuf;
    const unsigned char *p = buf, *e = buf + len;
    if (context->in_buffer) {
        if (context->in_buffer + len < 16) {
            /* Not enough for block, just pass into buffer */
            memcpy(context->buffer + context->in_buffer, p, len);
            context->in_buffer += len;
            return;
        }
        /* Put buffered data together with inbound for a single block */
        memcpy(context->buffer + context->in_buffer, p, 16 - context->in_buffer);
        MD2_Transform(context, context->buffer);
        p += 16 - context->in_buffer;
        context->in_buffer = 0;
    }
    /* Process as many whole blocks as remain */
    while ((p + 16) <= e) {
        MD2_Transform(context, p);
        p += 16;
    }
    /* Copy remaining data to buffer */
    if (p < e) {
        memcpy(context->buffer, p, e - p);
        context->in_buffer = e - p;
    }
}

void MD2Final(unsigned char output[16], void *contextBuf){
    MD2_CTX *context = (MD2_CTX*)contextBuf;
    memset(context->buffer + context->in_buffer, 16 - context->in_buffer, 16 - context->in_buffer);
    MD2_Transform(context, context->buffer);
    MD2_Transform(context, context->checksum);
    memcpy(output, context->state, 16);
}

void make_digest(char *md5str, const unsigned char *digest, int len) /* {{{ */
{
    static const char hexits[17] = "0123456789abcdef";
    int i;
    for (i = 0; i < len; i++) {
        md5str[i * 2]       = hexits[digest[i] >> 4];
        md5str[(i * 2) + 1] = hexits[digest[i] &  0x0F];
    }
    md5str[len * 2] = '\0';
}

void do_md2(char *arg){
    MD2_CTX context;
    unsigned char digest[16];
    md2str[0] = '\0';
    MD2Init(&context);
    MD2Update(&context, (unsigned char*)arg, strlen(arg));
    MD2Final(digest, &context);
    make_digest(md2str, digest, 16);
    Serial.print("MD2 value : ");
    Serial.println(md2str);
}




int CallBackBlockCount = 0; // counts the number of callbacks ( packets ) for each web page

static uint32_t timer;
const char website[] PROGMEM = "www.galatime.co.za";
boolean SDavailable;

char TXbuf[20]; // array to hold the query string sent to the URL ( the part after the  ?  in the URL )


void WriteSDfile(int WriteFileNo = 0, int RemFile = 0, char* wrTxt = ""){

    File fh;
    if(RemFile == 1){
        if (WriteFileNo == 1) SD.remove("file1.txt");
        if (WriteFileNo == 2) SD.remove("file2.txt");
        if (WriteFileNo == 3) SD.remove("file3.txt");
    } 
    if (WriteFileNo >= 1 && WriteFileNo <= 3){
        // open the file. note that only one file can be open at a time, so you have to close this one before opening another.
        if (WriteFileNo == 1) fh = SD.open("file1.txt", FILE_WRITE);
        if (WriteFileNo == 2) fh = SD.open("file2.txt", FILE_WRITE);
        if (WriteFileNo == 3) fh = SD.open("file3.txt", FILE_WRITE);

        if (fh) {       // if the file opened okay, write to it:
            fh.write(wrTxt);
            fh.close();             // close the file:
        } else {
            Serial.println("......................................");
            Serial.println("error opening SD file for writing");  // if the file didn't open, print an error:
            Serial.println("......................................");
        }
    }
}


void ReadSDfile(int ReadFileNo = 0){

    if (ReadFileNo >= 1 && ReadFileNo <= 3){
        File fh;
        Serial.println("......................................");
        Serial.print("SD Read File No : ");
        Serial.println(ReadFileNo);
        Serial.print("SD Available : ");
        Serial.println(SDavailable);
        Serial.println("......................................");
        if(SDavailable != false) {
            if (ReadFileNo == 1) fh = SD.open("file1.txt", FILE_READ);
            if (ReadFileNo == 2) fh = SD.open("file2.txt", FILE_READ);
            if (ReadFileNo == 3) fh = SD.open("file3.txt", FILE_READ);
            if(!fh) {
                        Serial.println("Error : SD open fail");
            } else {
                while(fh.available()) {
                    char ch = fh.read();                // read 1 char at a time from the SD file
                                        Serial.print(ch);  // just dump the file content to the serial monitor
                                        chx[0] = chx[1];
                                        chx[1] = chx[2];
                                        chx[2] = chx[3];
                                        chx[3] = ch;
                                        if(chx[0] == '<' && chx[1] == 'b' && chx[2] == 'r' && chx[3] == '>') Serial.println("");
                } // end of : while(fh.available)
                fh.close();
            }  // end of : if SD opened OK
        }else{
            Serial.println("Error : SD connect failed");
        }
    }
}



static void CopyXbuf(){
    // move each MD2 of the buffer stored in the xBuf arrays up 1 'line'
    memcpy( xBuf1, xBuf2, sizeof(xBuf2) );
    memcpy( xBuf2, xBuf3, sizeof(xBuf3) );
    memcpy( xBuf3, xBuf4, sizeof(xBuf4) );
    xBuf4[0] = 0;
}


static boolean ByteArrayCompare(char a[], char b[],int array_size){
    for (int i = 0; i < array_size; ++i)
      if (a[i] != b[i]) return(false); // if there is 1 character difference, return false = not a match
    return(true); // if false has not already been returned, then it must be true = arrays DO match
}

// called when the client request is complete
static void my_callback (byte status, word off, word len) {

    memcpy( ebuf2, Ethernet::buffer, sizeof(Ethernet::buffer) );  // immediately copy the ethernet buffer to ebuf2, in case the ethernet buffer is modified while my_callback is running
    ebuf2[off+len] = 0;  // after the offset and the length, add an end marker
    do_md2((char*)ebuf2 + off); // make an MD2 hash of the ebuf2 array

    // check if this buffer has already been received by comparing previous MD2s
    byte xBufExists = 0;
        if(ByteArrayCompare(xBuf1,md2str,33)) xBufExists = 1;
        if(ByteArrayCompare(xBuf2,md2str,33)) xBufExists = 2;
        if(ByteArrayCompare(xBuf3,md2str,33)) xBufExists = 3;
        if(ByteArrayCompare(xBuf4,md2str,33)) xBufExists = 4;

    if(xBufExists != 0){
        Serial.println("......................................");
        Serial.print("Received Buffer already exists - ignoring buffer match : ");
        Serial.println(xBufExists);
        Serial.println("......................................");
        IgnoreCount++;
    }else{
        CallBackBlockCount++;
        Serial.println("......................................");
        Serial.print("my_callback initiated. Block : ");
        Serial.println(CallBackBlockCount);
        Serial.println("......................................");
        CopyXbuf();  // move the MD2 values up a 'line'
        memcpy( xBuf4, md2str, sizeof(md2str) ); // add the current MD2 to the last array
        unsigned int bLen = len;  
        Serial.print("bLen = ");
        Serial.println(bLen);
        if(bLen < 512){
            Serial.println("End of web page -- Disabling Persistant Connection");
            ether.persistTcpConnection(false); // stop receiving more buffers
        }  
        Serial.print("Status : "); Serial.print(status); Serial.print("  :   off : "); Serial.print(off); Serial.print("  :   len : "); Serial.println(len);
        Serial.println((const char*) ebuf2 + off);
        WriteSDfile(2,0,((char*) ebuf2 + off));


//delay(500);  // uncomment this line will probably slow the callback function enough to cause multiple duplicate buffer packets to occur.


        if(bLen < 512){   // if this is the end of the page, then read and echo the SD file to the Serial Monitor
            Serial.println("......................................");
            Serial.println("Reading Contents of File 2");
            Serial.println("......................................");
            ReadSDfile(2);
            Serial.println(""); 
            Serial.println("......................................");
            Serial.print("Callback Block Count = "); Serial.println(CallBackBlockCount);
            Serial.print("Loop Count = "); Serial.println(LoopCount);
            Serial.print("Ignore Count = "); Serial.println(IgnoreCount);
            Serial.println("......................................");
        }   
    }
}




void SendDataToServer(char* SendTxt) {
    Serial.println("Browse Started");
    ether.persistTcpConnection(true);
    ether.browseUrl(PSTR("/inc/ArdEvents.php?"), SendTxt, website, my_callback);
}



void setup () {

    Serial.begin(57600);

    pinMode(53, OUTPUT);
    digitalWrite(53, HIGH);  // set pin 53 as Output and High otherwise SPI will not work

    SDavailable = SD.begin(42);  // set the CS pin number for the SD card here
    Serial.println("SD Card Started");
    Serial.print("SD Available = ");
    Serial.println(SDavailable);

    if (ether.begin(sizeof Ethernet::buffer, mymac, 46) == 0){   // set the CS pin number for the ethernet module here
        Serial.println(F("Failed to access Ethernet controller"));
    }else{
        ether.staticSetup(static_ip,static_gw,static_dns);
        Serial.println("Ethercard Started");
    }

    ether.printIp("IP:  ", ether.myip);
    ether.printIp("GW:  ", ether.gwip);  
    ether.printIp("DNS: ", ether.dnsip);  

    if (!ether.dnsLookup(website)) Serial.println("DNS failed");
    ether.printIp("SRV: ", ether.hisip);

} // end of : Setup


void loop () {

    ether.packetLoop(ether.packetReceive());  // this line must be in the start of the loop

    if(LastMil == 0 || (millis() - LastMil) > 30000){  // run this code during the first loop, and then every 30 seconds
        LastMil = millis();
        LoopCount++;
        Serial.print("Loop Count = ");
        Serial.println(LoopCount);
        CallBackBlockCount = 0;
        SD.remove("file2.txt");  // remove the file from the SD card
        sprintf(&(TXbuf[0]), "editgala=%d", 5);  // data to append to the URL to pass to the php script
        SendDataToServer(TXbuf);  // get the data from the php page
    }
}

RE: Resolved: Does ethercard browseurl callback duplicate callback with persistTcpConnection ? - Added by blondak almost 4 years ago

I found another solution

in callback function

uint32_t NextSeq = 0;

uint32_t get_seq() {
    return (((uint32_t)Ethernet::buffer[TCP_SEQ_H_P]*256+Ethernet::buffer[TCP_SEQ_H_P+1])*256+Ethernet::buffer[TCP_SEQ_H_P+2])*256+Ethernet::buffer[TCP_SEQ_H_P+3];
}

void users_push_callback(uint8_t status, uint16_t off, uint16_t len) {
    if ((NextSeq == 0) || (get_seq() == NextSeq) ){
        NextSeq = get_seq() + len;
    } else {
        return;
    }

    ....
}

    NextSeq = 0;
    ether.persistTcpConnection(true);
    ether.browseUrl(,....users_push_callback);

It works only for TCP in header of callback function I compute next TCP sequenceID and compare with ID of recieved packet, if IDs doesnt match they will be retransmitted ba TCP stack and packets will get in right order,… It seems to work at least for me,…

PS: get_seq is copied from tcpip.c and it should be make as public or this check can be “hardcoded” in tcpip.c EtherCard::packetLoop

    (1-4/4)