Decoding RF12 packets (in Python..)

Added by Anonymous 11 months ago

hi all,

what I thought would be a simple task is driving me nuts, how to interpret the data the rf12demo writes to serial.
For example, this line:

OK 23 34 193 3

while on the sending side I know these two 10bit integers were sent:

290 240

Now I know about the posts from jcw on the topic, someone kindly gathered them here: http://www.thebackshed.com/forum/uploads/DuinoMiteMegaAn/2011-12-24_041032_MSPNode_JeeNode_Protocol_R2.pdf
That also tells to expect little endian, but I'm not sure how that applies to bitfields.
And in anyway I'm not very comfortable with this level of software writing.

If I simply concatenate the binary values for each of the bytes and then decode, I get nonsensical values.
But I don't know what I could be doing to change the order of the values or even reversing the bits to account for endianness.

There are various workarounds but I'd like to get this right.

With the above example, I know theres 2x10 bits, so I take the rf12demo printout to be 2.5 bytes, read in binary:

('00100010', '11000001', '0011')

But reading exactly half of it to decimal integer does not reproduce the expected value, neither the second part.
Also reversing of the values or reordering of them where to no avail, although I'll admit I'm ignorant of handling endianness.

Searching github did turn up a project JeeHandler, but that seems to have settled on using 2-byte integers and byte-multiples.


Replies (16)

RE: Decoding RF12 packets (in Python..) - Added by JohnO 11 months ago

I decode 16 bit integers using bash. I don't think my head could get around 10 bits.

// args=("$@")    # Assign to array
//      17 42 40 0 65 0 37 150
//  args 0  1  2 3  4 5  6   7  
 let "node = ${args[0]}"    # Get node number
 let "sequence = ${args[1]}"    # Get sequence sequence number  
 let "count = ${args[3]}" 
 let "count = $count << 8" 
 let "count = $count + ${args[2]}" 
 let "timer = ${args[5]}" 
 let "timer = $timer << 8" 
 let "timer = $timer + ${args[4]}" 
 let "timer = $timer / 1000"    # Reduce to seconds 
 let "temperature = ${args[6]}" 
 let "temperature = $temperature - 8"    # Calibration   
 let "voltage = ${args[7]}" 
 let "voltage = $voltage * 20" 
 let "voltage = $voltage + 1000"       

RE: Decoding RF12 packets (in Python..) - Added by JohnO 11 months ago

I guess we need to do the high/low byte interchange on each pair to build up the string and then shift out 10 bits at a time to recover your values.

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

Yes, I remember AVR using words of two bytes. jcw's article gives too the impression this applies.

So swapping two bytes around before decoding makes sense, at least thats better than random tries :)

Playing with bitshifting now while time allows..

Here's a previous attempt, very naive but I'd like to understand basic first.
Swapping two bytes around though does not seem to be right.

#34 193 3 = 290 240

b = "".join((
    bin(34)[2:].rjust(8, '0'),
    bin(193)[2:].rjust(8, '0'),
    bin(3)[2:].rjust(8, '0'),
))

print int(b[0:10], 2)
print int(b[10:], 2)

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

Hmm... think I got it

I just tried all possibilities, manageable though not very intelligent :D

This prints out the correct values, now I need to make sense of it and use a more compact approach. Will get back later.


b = "".join((
    bin(3)[2:].rjust(4, '0'),
    bin(193)[2:].rjust(8, '0'),
    bin(34)[2:].rjust(8, '0'),
))

print int(b[0:10], 2) # 240
print int(b[10:], 2) #290

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

Here's the algorithm I use now:

- Get the binary notation for each byte
- If needed, padd each binary number to 8. Adjust the last byte to account for exact package length.
- Now reverse this list of binary byte values, and concatenate to one bitstring.
- Now, again in reverse, read in each bitfield and decode. Iow. starting with the bits for the last field, second-last, etc.

I'm not sure if there's an easier way using bitshifting, because its not clear across which transmission bytes the actual values are stretched, it needs something adaptive.
I guess I'd also run into sys.maxint when I try bitshifting from one large integer value.
So that definitely needs something smarter than I can write or need for now.

I was thinking about bash, really have no clue how to deal with integer and bytes there.

""" 

""" 
import bitstring

# Payload bitfields: LDR, DHT11 rhum, DHT11 temp, ATmega temp, lowbat
payload_len = 8 + 10 + 10 + 10  + 1
print 'Packet length', payload_len

payloads = (
        "0 24 193 163 1", # 0 280 240 26 0
        "0 22 205 163 1", # 0 278 243 26 0
    ) 
print payloads

payloads_bin = [
        [ 
            bin(int(b))[2:] for b in p.split(' ')
        ] for p in payloads
    ]
for i, p in enumerate(payloads_bin):
    # First padd all bytes, according to exact payload_len
    for x, p in enumerate(payloads_bin[i]):
        padd = 8
        remaining = payload_len - ( x * 8 )
        if remaining < 8:
            padd = remaining 
        payloads_bin[i][x] = payloads_bin[i][x].rjust(padd, '0')
    # reverse this sequence
    payloads_bin[i].reverse()

for b in payloads_bin:
    bits = ''.join(b)
    a = bitstring.ConstBitStream(bin='0b'+bits)
    # Now read payload fields in reverse
    print  \   
        a.read("bool"), \
        a.read("int:10"),\
        a.read("int:10"),\
        a.read("int:10"),\
        a.read("int:8")

It seems using any library to account for endianness results in the need to work with whole-byte values so I didn't went there either.

Its perhaps a bit tedious, and I have not bumped in to max. payload length, but it was worth it.
Now I don't need payload knowledge at my Link node, and can simply retain a register of my nodes w/ metadata on the host pc.

btw, I looked a bit in housemon, but I could only find there are some integers emitted for each rf12 packet, not the decoding part.
Is there a nice way in JS to do this?

RE: Decoding RF12 packets (in Python..) - Added by juewis 11 months ago

Hi mpe,
i'm using this script to decode the payload from my sensors:

# -*- coding: utf-8 -*-
from serial import *
import io
import struct
import binascii
# import signal
# import sys
# def signal_handler(signal, frame):
#     print 'You pressed Ctrl+C!'
#     sys.exit(0)
# signal.signal(signal.SIGINT, signal_handler)

#12880000005DE10099B95040
#114C0100007366661E3C00000000
#126400000073E400C9F6E33F6001
#121F000000740000CC3836409B003333534000000000000000000080CD41
#IDP I N G VChhhhffffffffhhhhffffffffLLLLLLLLLLLLLLLLffffffff
#1003000000730000D4814240A500A8C65340

DEBUG = False

def decode(values):
    _celsius = 0
    if len(values) == 24: #short line
        _id, _seq, _vcc, _water, _vbat = struct.unpack('<BLBHf',binascii.unhexlify(values))
        _vcc = _vcc * 0.02 + 1
        _water = _vcc / 1024 * _water
        _aread = 0
    elif len(values) == 28:
        _id, _seq, _vcc, _water, _vbat, _aread= struct.unpack('<BLBHfH',binascii.unhexlify(values))
        _vcc = _vcc * 0.02 + 1
        _water =  _water * 0.02
    elif len(values) == 36:
        _id, _seq, _vcc, _water, _vbat, _aread, _vccread = struct.unpack('<BLBHfHf',binascii.unhexlify(values))
        _vcc = _vccread
        _water =  _water * 0.02
    elif len(values) == 60:
        _id, _seq, _vcc, _water, _vbat, _aread, _vccread, _a, _b, _celsius = struct.unpack('<BLBHfHfLLf',binascii.unhexlify(values))
        _vcc = _vccread
        _water =  _water * 0.02
    else:
        _id, _seq, _vcc, _water, _vbat = struct.unpack('<BLBff',binascii.unhexlify(values))
        _aread = 0
        _vcc = _vcc * 0.02 + 1

    print "ID=%s SEQ=%s VCC=%.3f BATTERY=%.3f WATERSENSOR=%s TEMP=%.3f AREAD=%s" % (_id, _seq, _vcc, _vbat, _water, _celsius, _aread)

ser = Serial(port='com9',baudrate=57600,timeout=4)
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser))
while True:
    try:
        line = sio.readline()
        if len(line) > 0:
            if DEBUG: print "<%s>:%s" % (len(line),line)
            if len(line) > 2:
                if line[1] == 'E':
                    ser.write('1x')
                    ser.write('1q')
                if line[:3] == 'OKX':
                    if DEBUG: print len(line)
                    if len(line) == 29:
                        _values = line[4:28]
                    elif len(line) == 33:
                        _values = line[4:32]
                    elif len(line) == 41:
                        _values = line[4:40]
                    elif len(line) == 65:
                        _values = line[4:64]
                    else:
                        _values = line[4:32]
                    decode(_values)
    except KeyboardInterrupt:
        ser.close()
        break

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

Thanks for the tip. But these are whole byte values, right.
LE value decoding works with discrete byte values.
Ie. H is a two-byte unsigned, etc. http://docs.python.org/2/library/struct.html

But the JeeLib examples I've seen use bitfields, for example:

struct {
    byte light;     // light sensor: 0..255
    byte moved :1;  // motion detector: 0..1
    byte humi  :7;  // humidity: 0..100
    int temp   :10; // temperature: -500..+500 (tenths)
    byte lobat :1;  // supply voltage dropped under 3.1V: 0..1
} payload;

So this is 8 + 1 + 7 + 10 + 1 = 27 bits, ie. between 3 and 4 bytes.

This is easily decoded in C++ by casting the values into the struct on the receiving JeeNode, but I would like to keep node and payload types out of the central Link node.

Btw. I'm using LineReceiver from Twisted protocols to implement the serial reader.

Anyway, thanks for your comment!

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

btw, I looked a bit in housemon, but I could only find there are some integers emitted for each rf12 packet, not the decoding part.
Is there a nice way in JS to do this?

Apropos, any word on how/where Housemon does this, or is that doing whole bytes too?

RE: Decoding RF12 packets (in Python..) - Added by juewis 11 months ago

I would imply that some additional bit shifting is required in your case.
I'm not familiar with the Housemon solution, because i plan to stick with python and django as central server component in my monitoring solution.

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

I would imply that some additional bit shifting is required in your case.

Where? I cannot use struct.unpack, but would need to do it all manually. So yes, I could use bitshifting if I'm careful about max-int, endianness etc.

Also this is not 'my case', the last example I gave is standard JeeLand stuff.

RE: Decoding RF12 packets (in Python..) - Added by juewis 11 months ago

i wasn't aware of the bitstring module.
I modified my previous script, which should now also be usable for the jeelib sample.

# -*- coding: utf-8 -*-
from serial import *
import io
import struct
import binascii
from bitstring import *

#12880000005DE10099B95040
#114C0100007366661E3C00000000
#126400000073E400C9F6E33F6001
#121F000000740000CC3836409B003333534000000000000000000080CD41
#IDP I N G VChhhhffffffffhhhhffffffffLLLLLLLLLLLLLLLLffffffff
#1003000000730000D4814240A500A8C65340

# id byte by jeelib
# typedef struct {
#   long ping;           // 32-bit counter
#   byte vccbyte;        // vcc voltage
#   uint16_t waterlevel; // waterlevel sensor
#   double battery;      // battery voltage
#   uint16_t anaread;
#   double vccread;      // measured vcc
#   byte sensoraddr[8];  // address of ds18b20
#   double celsius;      // temperature reading in Celsius
# } record;

DEBUG = False

def convert2bits(payload):
    """convert decimal string back to bitstream
    and to hex string
    """ 
    bits = ''.join([chr(int(b)) for b in payload.split()])
    hex = '0x'+binascii.hexlify(bits)
    return hex

def decode(hex):
    """ decode hex string to attributes
    """ 
    a = BitStream(hex)
    return a.read("uint:8"), \
           a.read("uintle:32"), \
           a.read("uint:8"), \
           a.read("uintle:16"), \
           a.read("floatle:32"), \
           a.read("uintle:16"), \
           a.read("floatle:32"), \
           a.read("hex:64"), \
           a.read("floatle:32")

def report(hex):
    _id, _seq, _vcc, _water, _vbat, _aread, _vccread, _a, _celsius = decode(hex)
    print "ID=%s SEQ=%s VCC=%.3f BATTERY=%.3f WATERSENSOR=%s TEMP=%.3f AREAD=%s" % (_id, _seq, _vccread, _vbat, _water, _celsius, _aread)

ser = Serial(port='com9',baudrate=57600,timeout=4)
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser))
while True:
    try:
        line = sio.readline()
        if len(line) > 0:
            if DEBUG: print "<%s>:%s" % (len(line),line)
            if len(line) > 2:
                if line[1] == 'E':
                    ser.write('1q')  # suppress bad packets
                if line[:2] == 'OK':
                    report(convert2bits(line[2:])) # split off "OK" 
    except KeyboardInterrupt:
        ser.close()
        break

RE: Decoding RF12 packets (in Python..) - Added by juewis 11 months ago

I have done some more investigation on the topic and inspired by the article of jcw i did some tests.

The above script is not usable on bitfields as is.
I made the following observations:
The bitfield as defined in a struct is packed in LE byte order and transmitted in this order.
The individual bitfields are defined from bottom to top of the struct.

struct {
    byte light;     // light sensor: 0..255
    byte moved :1;  // motion detector: 0..1
    byte humi  :7;  // humidity: 0..100
    int temp   :10; // temperature: -500..+500 (tenths)
    byte lobat :1;  // supply voltage dropped under 3.1V: 0..1
} payload;

In the above example the bitfield is a byte array of three bytes filled with lobat first, then temp, humi, moved.
The bitfield uses 19 bits, so there are 5 bits unused.
payload.light is not part of this bitfield, but a separate byte.
The bitfield is transmitted in LE byte order and received by RF12Demo as 3 decimal numbers.
These are stored in the payload string in the sample.

# -*- coding: utf-8 -*
from bitstring import *
import binascii
import struct

# struct {
#     byte light;     // light sensor: 0..255
#     byte moved :1;  // motion detector: 0..1
#     byte humi  :7;  // humidity: 0..100
#     int temp   :10; // temperature: -500..+500 (tenths)
#     byte lobat :1;  // supply voltage dropped under 3.1V: 0..1
# } payload;

#   payload.light = 125;
#   payload.moved = 1;
#   payload.humi = 63;
#   payload.temp = -420;
#   payload.lobat = 1;

payload = "127 92 6" 

def convert2hex():
    """convert decimal number back to hex string
    bitfields are packed with little endian byte order
    reverse the number before converting
    """ 
    numbers = [b for b in payload.split()]
    numbers.reverse()
    bits = ''.join([chr(int(b)) for b in numbers])
    hex = '0x'+binascii.hexlify(bits)
    print hex
    return hex

bits = BitStream(convert2hex())
print bits.read("bin:24")

a = BitStream(convert2hex())
"""read bitstream in reverse order as defined in struct
""" 
aa = a.readlist("bits:5, bits:1, bits:10, bits:7, bits:1")
aa.reverse()
print aa[0].read("uint:1") # payload.moved
print aa[1].read("uint:7") # payload.humi
print aa[2].read("int:10") # payload.temp
print aa[3].read("uint:1") # payload.lobat
print aa[4].read("uint:5") # unused bits of payload

Running the script gives the following output:

0x065c7f
000001100101110001111111
0x065c7f
1
63
-420
1
0

When mpe started the thread he had a problem with two 10 bit int fields.

# -*- coding: utf-8 -*
from bitstring import *
import binascii
import struct

# struct {
#     int temp1   :10; // temperature: -500..+500 (tenths)
#     int temp2   :10; // temperature: -500..+500 (tenths)
# } payload;

#   payload.temp1 = 290;
#   payload.temp2 = 240;

payload = "34 193 3" 

def convert2hex():
    """convert decimal number back to hex string
    bitfields are packed with little endian byte order
    reverse the number before converting
    """ 
    numbers = [b for b in payload.split()]
    numbers.reverse()
    bits = ''.join([chr(int(b)) for b in numbers])
    hex = '0x'+binascii.hexlify(bits)
    print hex
    return hex

bits = BitStream(convert2hex())
print bits.read("bin:24")

a = BitStream(convert2hex())
"""read bitstream in reverse order as defined in struct
""" 
aa = a.readlist("bits:4, bits:10, bits:10")
aa.reverse()
print aa[0].read("int:10") # payload.temp1
print aa[1].read("int:10") # payload.temp2
print aa[2].read("uint:4") # unused bits of payload

Output when running the script:

0x03c122
000000111100000100100010
0x03c122
290
240
0

Decoding RF12 packets (in Python..) - Added by jcw 11 months ago

The individual bitfields are defined from bottom to top of the struct.

Very odd. When you look at the OK packet from RF12demo, then the first byte has the light intensity, the +moved+humi packed into one byte, etc.

Are you sure you're looking at it in little-endian order?

(yeah, this stuff is always very confusing... so many ways to "interpret" what you see)

RE: Decoding RF12 packets (in Python..) - Added by juewis 11 months ago

The light intensity is one byte.
The remaining 4 variables (19 bits) are packed into three bytes (24 bits).
RF12Demo receives OK 125 127 92 6.
125 is decoded as byte.
The next three values are reversed (6 92 127), concatenated as bytes and converted to a hex string. (0x065c7f)
This string as bit representation is 000001100101110001111111.
String separated into the individual fields: 00000 1 1001011100 0111111 1 (unused,lobat,temp,humi,moved).

RE: Decoding RF12 packets (in Python..) - Added by Anonymous 11 months ago

@jeuwis

hmmm.. as I see it, you are printing the bitstring in reverse. As I did, you reverse at two points to compensate for lack of endianness in libs while using it to decode bitfields to primitives. This is why it seemed to me python stdlib and others do not do bitfield decoding.

String separated into the individual fields: 00000 1 1001011100 0111111 1 (unused,lobat,temp,humi,moved).

I have not run it but I think to see this prints between the reversals.
Here is what I get for your example values (1, 63, -420, 1) setting light at 0 :

MSB  LSB
00000000
01111111
01011100
00000110

My script again: http://pastebin.com/T5ctzwzp

This is the map I just made for the RoomNode payload:

         MSB  LSB
 Byte 0  76543210   Light; bit 7-0
 Byte 1  65432100   Humi; bit 6-0, moved; 1 bit
 Byte 2  76543210   Temp; bit 7-0
 Byte 3  -----098   (5 bits unused), Lobat; 1bit, Temp MSB's 9-8

Granted, this just prints the struct and does not sent it over RF12, but it seems to me that is what is happening.

I'm getting nice exercise in thinking in binary encoded values and representing them in bytes and integers and such :)
And I see how to go about using bitwise operations now too, thanks for all the examples all.

RE: Decoding RF12 packets (in Python..) - Added by Remcokortekaas 10 months ago

Inspired by the Mosquitto Parser Python code of Kyle Gordon (who uses a non-standaard roomnode struct):
This decodes the standard roomnode struct:

        msg = ser.readline()
        data = msg.split(" ")
        a   = int(data[2]) 
        b   = int(data[3])
        c   = int(data[4])
        d   = int(data[5])       
        node_id = str(int(data[1]) & 0x1f)
    msg_seq = str(int(data[3]))
        light       = a 
        motion      = b & 1
        humidity    = b >> 1
        temperature = str(((256 * (d&3) + c) ^ 512) - 512)
        battery     = (d >> 2) & 1
        temperature = temperature[0:2] + '.' + temperature[-1]

(1-16/16)