Project

General

Profile

Intermitent reception problems with JeeNode / RFM69CW

Added by fgomes almost 4 years ago

I have 2 JeeNodes with RFM69CW 915MHz from Moderndevice.com. One is simply transmitting sensor data every 5 seconds. The other is receiving this data and updating an LCD with it (and reporting by the serial port). No Acks are being used in this test. I have the receiver permanently powered, sending messages every 5 seconds. I’m finding that when I power on the receiver it receives a few messages from the transmitter, but then stops receiving more messages (the remaining functions still work, but doesn’t receive radio messages). If I restart the receiver it receives again some messages and then stops receiving more messages. I’ve another node acting as a sniffer and it receives every message during the test, so it confirms that the transmitter is transmitting. I’ve also found that when i left the receiver powered, it is able to receive some messages from time to time, for example during one test it received 5 consecutive messages when I powered it on, then it didn’t receive any message for 50 minutes (the transmitter is transmitting every 5 seconds), then it recovered and received a few more messages (17 consecutive messages). The RSSI in the receiver node when it receives is always very good, the nodes are physically close (RSSI between –27 and –37 dBm).

This occurred in the past with a node that was replaced and the problem was solved, so I’ve considered that it was a problem with that radio, but now it is occurring again with a new module I’ve just received, I’m starting to get worried about that. Do you have some clue to what could be the problem? Some additional tests I could do in order to figure the root cause of this problem?

Thanks!

Fernando


Replies (35)

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

I had thought that we had cured the master branch of this issue. Could you try using the https://github.com/jcw/jeelib/tree/RFxConsole branch from github?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

Hi JohnO, thanks for your advice! I’ve changed the jeelib to the one you mentioned (I have already used it for the RFM69HW support), and now it doesn’t block anymore! Thanks!

I noticed an intermediate problem when I only used your recommended jeelib version on the receive node (keeping the transmitting node as it was, with the standard jeelib version): the transmitting node sends a message sequence number in each message, together with the sensor data, and in the receive node using the new library it only receives the messages with odd sequence number (1, 3, 5, …). This was also corrected when i flashed the transmitting node using the same Jeelib version you suggested.

Can you point me to the reason of the previous problem - the receive node block after some messages? If there is a forum discussion or something that could give me an idea of the problem. In the meantime if you want me to make any additional test in order to clarify the issue please let me know, I’m here to help!

Best regards

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

I’m using the library branch suggested by JohnO in the previous post (https://github.com/jcw/jeelib/tree/RFxConsole), and I’m still getting blocking situations, but now only after long time (with the standard library it blocked after a few messages). The setup was one node sending with acknowledgement, one node receiving and making the acknowledgement, and a third node only sniffing, near the receiving node. The transmitting and receiving nodes are about 8 meters apart, with 3 walls between them. The transmitting node sends a message every 2 seconds. After some thousand messages transmitted (some hours transmitting) the receiving node stops receiving (and acking, of course), but the sniffer node that is right on its side keeps receiving without any problems. The receiving node keeps all other functions working, like updating its LCD, talking by the serial port, so the ATMEGA is not blocked, only rf12_recvDone() never returns true anymore. If I reset the receiving node all returns to work, so it seems the RFM69 radio is blocked somehow. I’ve also observed that the last message received by the receiving node before blocking this last time had a much lower RSSI seen by the receiving node, but the sniffer kept the same RSSI level (both were observing RSSI levels around –70dBm, and the receiving node reported –90dBm in last message before blocking). This RSSI observation was done one time only so I don’t know if this is systematic, I’m waiting for it to block again and update here later.

Any tip on how to diagnose this problem? Is it possible to reset the radio (RFM69), like calling the rf12_initialize function again if it gets blocked?

Thanks!

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

An interesting development. May I ask if the sniffing node is also RFM69 equipped or is it using RFM12B?

I have seen the blip in RSSI value when a spike of noise drags the receiver away a little from the frequency that the transmitter is operating on. You are using your own code I guess. There are a whole raft of values stored by the RFxConsole library, you could start with:

        observedRX.afc = (RF69::afc);                  // Grab values before next interrupt
        observedRX.fei = (RF69::fei);

        observedRX.lna = (RF69::lna >> 3);

The above together with the RSSI value can reveal quite a lot. AFC is automatic frequency control, FEI is frequency error indicator and LNA the low noise amplifier setting.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

I also suspect that if you forced the receiving node to transmit a packet all would return to normal.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

Thanks JohnO!

I’ve added the print of the AFC, FEI and LNA to the RSSI print after each received message. Also added an extra attempt to recover sending a message from the receiving node when it doesn’t receive for a long time. For now it still didn’t block, I’ll post here the log when it does. Currently I’m getting these values after each received message:

RSSI: –73
AFC: 0
FEI: 230
LNA: 24

I’ve observed that the LNA is always 24, the FEI has changed between –456 and 331 and AFC is mainly 0, but has some different values from time to time, I’ve seen 222, 102, but only a few times.

When I get a blocking situation I’ll report here the results.

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

The LNA needs shifting right 3 positions to obtain a value between 0 and 7. Typically it will be quite stable when it become unstable is when problems start to happen. 24 >>3 is I think a value of 3 which is bang in the middle of the range and fine.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

It has already blocked, with the following values in the last reception:

RSSI: –81
AFC: 0
FEI: 255
LNA: 16

Before reaching this situation it started receiving with lower RSSI values like you can see in the attached log (starting with the last ‘normal’ reception value.

I put the receiving node sending a message when it blocks, and i can see it in the sniffer but unfortunatly I’ve a bug in the code because it is permanently sending a message, and it doesn’t recover the reception, I’ll correct that to see if it works.

Does the log give some clue? The LNA changed toghether with the RSSI change.

Thanks again!

Fernando

rxlog.txt (10.8 KB) rxlog.txt

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

LNA of two looks fine, it is amplifying the received signal a little more than before but still not crazy. Do you know if you are transmitting at full power with the default noise floor?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

Temperature drop of two and a half degrees?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by jcw almost 4 years ago

Wild guess: is the voltage regulator overheating and then switching off?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

Thanks for your help!

JohnO: are you refering to the ‘temperature’ information included in the log? This is just ‘noise’, I don’t have the temperature sensor in the transmitting node, so it is an analog input floating.
JCW: Are you refering the JeeNode voltage regulator in the receiving node? The JeeNode voltage regulator seems to be ok, since all the node functions are still working. The transmitting node is also ok because i continue to see the messages in the sniffer node.

What is consistent is that every time it fails the RSSI drops before it happens. It happened again, and transmitting a message seems to help, but didn’t solve the issue, it started working, but with a lower RSSI (lower than –80dBm, where the sniffer keeps indicating the same –73dBm right on its side) and fails intermittently (recovering when transmitting a message). If I reset the receiving node the RSSI goes again to –70dBm… For example, now, after recovering sending a message, this is what I have:

RSSI: –82
AFC: 0
FEI: 151
LNA: 2

Any clues?

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

I forgot to answer to JohnO question about transmitting power, I don’t change any setting for that. Here are the main pieces of code related with the radio:

setup:

rf12_initialize(18, RF12_915MHZ, 212);

loop:

if (rf12_recvDone()) {
if(rf12_crc == 0) {
lastRX = lastRecover = millis();
last_rssi = -(RF69::rssi>>1);
last_afc = (RF69::afc);
last_fei = (RF69::fei);
last_lna = (RF69::lna >> 3);
processData();
// Ack
if(RF12_WANTS_ACK) {
replyMsg.currentTime = millis();
rf12_sendStart(RF12_ACK_REPLY, &replyMsg, sizeof(replyMsg));
}
}

if((millis()-lastRecover) > RECOVER_TIMEOUT) {
Serial.print(“Trying to recover RFM69”);
replyMsg.currentTime = millis();
rf12_sendNow(0, &replyMsg, sizeof(replyMsg));
rf12_sendWait(0);
lastRecover = millis();
}

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

I think the –80dB is very close to the configured noise floor. What hardware is your sniffer using?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

In RF69.cpp is a line:

  0x29, 0xA0, // RssiThresh ... -80dB

Try changing this to 0xB0, // giving –88dB

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

I’ll try changing the noise floor to –88dBm, nevertheless the strange thing is the signal level reduction observed in the receiver node, that returns to normal when I reset the receiver node. The sniffer is equal to the receiver, both are jeenodes SMD with rfm69 radios. And in the receiver node the signal level keeps good.

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

I think that the receivers AFC has moved the receive frequency to a non-optimal place. This drops the perceived signal below the noise threshold and the next transmission is not heard so the AFC doesn’t swing back. I would like to hear from martynj on his take on this one.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

Adding some additional information

After the Receiver is in this state (recovering during some time when sending a message) the values are the following:

RSSI: –85
AFC: 0
FEI: –334
LNA: 2

I’ve reset the receiver Jeenode, it got a bit better receiving messages, but still not very good RSSI:

RSSI: –82
AFC: 0
FEI: –334
LNA: 2

After that, I have powered down the receiver for 5 seconds and reconnected it again, now the values are the normal ones:

RSSI: –73
AFC: 0
FEI: 0
LNA: 3

after 10 minutes still working well, the values are the following:

RSSI: –73
AFC: 0
FEI: 241
LNA: 3

I will use SDR# in order to try to figure out if there is a center frequency variation in time.

It is also strange that the sniffer node (equal hardware, same location) never failed, it is working for days without a reset or power cycle). Could be a hardware problem with the receiver node? The hardware was supplied for both nodes by moderndevice at the same time.

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

Does the sniffer report constant RSSI etc at the same time as the receiver values decay?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

JohnO, yes, the sniffer stays always around –70dBm. Both nodes are at the same place, at about 8 meters from the transmitter. I never restarted the sniffer for about 2 days (since I made the last change to it, adding the RSSI information). I will swap the code between the sniffer and the receiver to check if there is some difference. The problem is that this isn’t deterministic, it might occur in 10 minutes or in 10 hours (for example I left it working yesterday night, this morning it was still working and failed after about 10 hours). It failed some minutes ago, after about two and half hours and this time the last RSSI was not so low as usual, and took a long time to recover with transmitting a message. The message it tries to transmit to recover is transmitted because the sniffer sees it

This was for the last message received before blocking (2:36 after power up):

Time = 9369608
RSSI: –76
AFC: 0
FEI: –314
LNA: 3

If I go a bit early in the log, the data was the following (2:34 hours after power up):

Time = 9263657
RSSI: –74
AFC: 0
FEI: –143
LNA: 3

And even earlier (2:33 hours after power up), it was just like after powering it up:

Time = 9197688
RSSI: –75
AFC: 0
FEI: 211
LNA: 3

After a long time blocked, it recovered (2:53 hours after power up):

Time = 10415207
RSSI: –74
AFC: 0
FEI: 550
LNA: 3

Now, 3:00 hours after power up, it is:

Time = 11000920
RSSI: –80
AFC: 0
FEI: –416
LNA: 3

So the FEI was 211 for about 2:30 hours and then it started to drop, ending at about –314 when it blocks, 6 minutes after started dropping the FEI. Since this indicates a frequency error, it is a bit strange that it was stable for a long time and then started to change, with everything else not changing. After recovering, the FEI was 550 but seven minutes later was –416. All the other monitored values didn’t change (AFC=0, LNA=3).

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by fgomes almost 4 years ago

To try to identify the difference between the sniffer and the receiver, I’ve exchanged the code between them, and here are my findings:

The node now running the sniffer (previously running the receiver in the above tests) is receiving with the same signal strength since I have powered it up (as the previous node running the sniffer), it is still receiving at between –68 and –74 dBm, similar to the previous node with the sniffer code.

The node now running the receiver (previously running the sniffer) started with a similar RSSI (about –70dBm) but started dropping the RSSI and blocked in less than 1:30 hours. Transmitting a message partially recovers it, but with lots of missed messages, at reception levels of about –80 dBm or less, while the sniffer keeps seeing the same transmissions at about –68dBm. So it seems to be related with the code, not with the hardware, perhaps some radio functions used in the receiver sketch trigger the issue. The only difference I saw is that the receiver sends acks to each received messages, and the sniffer only receives, never transmits any message. Could this give some clue to the problem?

Fernando

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

This is interesting and I think we need to fully understand it before resolving it. I take it that you have updated the RSSI threshold on both units to 0xB0 as previously mentioned.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

fgomes wrote:
> The only difference I saw is that the receiver sends acks to each received messages, and the sniffer only receives, never transmits any message. Could this give some clue to the problem?

There is a lot going on when the receiver turns around to transmit. What does the sniffer report about the ACK’s going back to the transmitter?

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

Perhaps we need to see the receiver code to appreciate exactly where the ACK is being sent.

RE: Intermitent reception problems with JeeNode / RFM69CW - Added by JohnO almost 4 years ago

Perhaps run a copy of RF12Demo as the receiver, including ACK’s and observe performance with that code. You may need to run RFxConsole.ino instead to have all the data dumped “0q”.

(1-25/35)