Before commenting read Forum rules
Don't comment the topic if you have a new question.
You can create a new topic selecting correct category from Gurux Forum and then create a new topic selecting "New Topic" from the top left.
Forums
Hi
Currently I'm trying to implement a tool based on your Push.Listener Example in Python that interprets the pushed unencrypted DLMS messages sent by a Iskraemeco AM550 meter. The received byte data looks promising and the Gurux DLMS Translator (GUI Tool) is able to parse the data and provide the containing COSEM objects with OBIS codes and values.
However, my tool and the Push Listener Example (see log below) cannot decode the received data and both end up with a "Invalid Command" Error.
I've already been doing some digging and found the problem. The data is coming in HDLC segments which can be seen at the beginning of the messages (7E A8 and last frame 7E A0). With an incomplete message (7E A0 frame not yet received), the function GXDLMS.getData(cls, settings, reply, data, notify) returns at line 1897. But when the whole message is finally received, GXDLMS.getDataFromFrame(reply, target, ...) only copies the last part of the message, i.e. the last detected frame of the whole reply buffer. Therefore, getPdu(settings, target) at line 1902 has not the full message content and fails to find the message type / command which leads to the Invalid Command error.
Do you have any idea how to circumvent this behavior or to fix the library code? We somehow need to get the content of all the individual frames and concatenate them to one buffer which is then parsed in the getPdu() function.
Just for additional information: I already successfully used my tool and the Push listener example with the L+G E450. There, the data is fragmented as well but on a different layer. All frames are complete (7E A0 beginning) and they use the "General Block Transfer" feature of DLMS to split up data into multiple frames/messages. The Gurux library handles this perfectly and enables the client to check if all data is sent with the isMoreData() method. Here, each message is fed into the getDataFromFrame() and getPdu() functions and therefore all data is available in the end.
I hope you can comprehend my description have a idea to fix the problem.
Thank you and best regards,
raymar
--------------------------------------------------------
Log Output of Push Listener Example:
gurux_dlms version: 1.0.104
gurux_net version: 1.0.17
gurux_serial version: 1.0.15
/dev/ttyUSB0:115200 8NONE1
Press any key to close the application.
Media state changed. MediaState.OPENING
trace:10:15:55 TraceTypes.INFO Settings: Port: /dev/ttyUSB0 Baud Rate: 115200 Data Bits: 8 Parity: Parity.NONE Stop Bits: 0 Eop:126
Media state changed. MediaState.OPEN
New data is received. /dev/ttyUSB0:7E A8 A4 CF 02 23 03 99 96 E6 E7 00 0F 00 00 03 89 0C 07 E4 08
New data is received. /dev/ttyUSB0:0A 01 0F 2A 0F 00 FF 88 80 02 1C 01 1C 02 04 12 00 28 09 06 00 06 19 09 00 FF 0F 02 12 00 00 02 04 12 00 28 09 06 00 06 19 09 00 FF 0F 01 12 00 00 02 04 12 00 01 09 06 00 00 2A 00 00 FF 0F 02 12 00 00 02 04 12 00 01 09 06 00 00 60 01 01 FF 0F 02 12 00 00 02 04 12 00 08 09 06 00 00 01 00 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 07 00 FF 0F 02 12 00 00 02 04 12 0E 87 7E 7E A8 A4 CF 02 23 03 99 96 00 03 09 06 01 01 03 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 04 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 00 FF 00 00 7E 7E A8 A4 CF 02 23 03 99 96 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 02 FF 0F 02 12 00 00 02 04 12 00 03 B2 E4 7E 7E A8 A4 CF 02 23 03 99 96 09 06 01 01 08 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 08 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 08 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 0D 07 00 FF 0F 02 12 00 00 09 06 00 06 19 09 00 FF 09 10 49 53 4B 31 30 33 30 37 37 35 32 31 33 38 35 39 09 07 31 38 37 36 33 35 30 09 0C 07 E4 08 0A 01 0F 2A 0F 00 FF 88 80 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 5F 0E A4 06 00 2F 44 2E 06 00 2F CA 76 06 00 00 FF CA 7E 7E A0 55 CF 02 23 13 A2 33 00 00 06 00 00 00 00 06 00 00 00 00 06 00 05 47 7C 06 00 03 84 3F 06 00 01 C3 3D 06 00 00 00 09 06 00 00 00 09 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 01 0A F7 06 00 00 94 B0 06 00 00 76 47 12 00 00 12 7B 7E
Invalid Command.
Hi,
Hi,
There is not all the data. I believe that the meter will send at least 4 frames and you have added only 3 frames. Can you add all the frames so I can check this? Because the first frame is missing parsing will fail.
BR,
Mikko
Hi Mikko
Hi Mikko
No, there are actually 5 HDLC frames in the provided data.
Here again the same data but split up into frames. The Gurux DLMS translator (from Gurux Director Version 8.2.2106.1501 (Windows)) can parse these and shows the data when selecting the option "Complete PDU".
7E A8 A4 CF 02 23 03 99 96 E6 E7 00 0F 00 00 03 89 0C 07 E4 08 0A 01 0F 2A 0F 00 FF 88 80 02 1C 01 1C 02 04 12 00 28 09 06 00 06 19 09 00 FF 0F 02 12 00 00 02 04 12 00 28 09 06 00 06 19 09 00 FF 0F 01 12 00 00 02 04 12 00 01 09 06 00 00 2A 00 00 FF 0F 02 12 00 00 02 04 12 00 01 09 06 00 00 60 01 01 FF 0F 02 12 00 00 02 04 12 00 08 09 06 00 00 01 00 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 07 00 FF 0F 02 12 00 00 02 04 12 0E 87 7E
7E A8 A4 CF 02 23 03 99 96 00 03 09 06 01 01 03 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 04 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 00 FF 00 00 7E
7E A8 A4 CF 02 23 03 99 96 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 05 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 02 FF 0F 02 12 00 00 02 04 12 00 03 B2 E4 7E
7E A8 A4 CF 02 23 03 99 96 09 06 01 01 08 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 08 08 01 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 08 08 02 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 0D 07 00 FF 0F 02 12 00 00 09 06 00 06 19 09 00 FF 09 10 49 53 4B 31 30 33 30 37 37 35 32 31 33 38 35 39 09 07 31 38 37 36 33 35 30 09 0C 07 E4 08 0A 01 0F 2A 0F 00 FF 88 80 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 5F 0E A4 06 00 2F 44 2E 06 00 2F CA 76 06 00 00 FF CA 7E
7E A0 55 CF 02 23 13 A2 33 00 00 06 00 00 00 00 06 00 00 00 00 06 00 05 47 7C 06 00 03 84 3F 06 00 01 C3 3D 06 00 00 00 09 06 00 00 00 09 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 01 0A F7 06 00 00 94 B0 06 00 00 76 47 12 00 00 12 7B 7E
Please try again with this representation.
Thank you for your quick response and your time!
Sincerely,
raymar
Hi,
Hi,
Thank you for this information. Now I know why this fails. This is working when the baud rate is slower, but in your case, all the bytes are read at once. This requires some changes.
This is fixed for the next release.
BR,
Mikko
Hi Mikko
Hi Mikko
Oh, wow, thank you!
Actually, I just tried to read in the serial data in small (10 Bytes) chunks. After every chunk I printed out the reply buffer (the raw data received from the serial interface) for visual verification, then called getData() and checked the notify object for completeness. This would simulate a slower baudrate - the data is not fed in all at once or in very large chunks.
I see how the buffer grows and isComplete() returns False until all data (5 HDLC frames) are read. Unfortunately, getData() then still raises a ValueError("Invalid Command") so it does not matter if the data is read in large or smaller chunks or all at once (also tested).
As I mentioned in the previous message, I assume the problem is a misbehavior when parsing data segmented on HDLC layer described in ISO13239, section 4.9.3. The library successfully detects an incomplete message and returns early, but does not utilize all data from the reply buffer when the message is eventually complete.
I'm looking forward to the next release and the fixes you mentioned.
Have a nice day!
Best regards,
raymar
Hi Mikko
Hi Mikko
You mentioned that you are going to fix the above problem in the new release. When are you planning to publish this release? We use the Gurux library in a project and this piece of information would be very helpful for planning on our side.
Concerning this, could you also release the new versions of the Python package on pypi.org (currently on 1.0.104)?
Thank you.
Best regards,
raymar
Hi,
Hi,
This is fixed and released to C#. We are waiting for community comments and if there is nothing special we try to release it for Python next week.
BR,
Mikko
Hi Mikko
Hi Mikko
I just wanted to ask what the status of releasing a new Python gurux-dlms version (1.0.106) on Github and pypi.org is. I'm still waiting for the fixes and recent improvements such that I run another test with the Iskra AM550 meter.
Thanks and best regards,
raymar
Hi,
Hi,
Our client has kept us busy and this was delayed because of that. The new version is on testing and is released latest tomorrow.
BR,
Mikko
Hi,
Hi,
This is fixed and the new version is released.
https://www.gurux.fi/node/19015
BR,
Mikko
Hello Mikko
Hello Mikko
Thank you very much for the new release! I tested it with the Iskra AM550 this morning and it worked indeed on the first try. The data can now be parsed with this meter.
However, now with the new version, unfortunately the data of another smart meter (L+G E450) cannot be parsed anymore which perfectly worked with the previous version 1.0.104.
I invested some time to find the actual problem. Probably, the following explanation could help you to fix the arisen issue:
The data received from my meters usually arrive frame by frame (7E .... 7E). In my wrapper code around the GXDLMSSecureClient.getData(reply, data, notify) I check the notify object with isComplete() and isMoreData(). Before, isMoreData() returned False exactly with the last frame of a complete DLMS object/packet, but now isMoreData() never returns False anymore and I therefore lose the information when a DLMS packet is complete and ready for parsing.
I traced the problem back to one code line introduced in the newest commit (or version 1.0.106):
https://github.com/Gurux/Gurux.DLMS.Python/commit/92ef27849665b132def3a…
on line 1908, you test the "target" object with isMoreData(). In the old release you checked the "data" object. I changed "target" to "data" and tested both smart meters again. With this minor modification the data of both meters can be parsed.
I'm not sure if this is just a bug or if indeed the target object must be checked for isMoreData and the actual problem is somewhere else.
Here is the data of a complete packet from the L+G meter which could be parsed with the previous version (or my fix above):
7E A0 84 CE FF 03 13 12 8B E6 E7 00 E0 40 00 01 00 00 70 0F 00 01 40 8A 0C 07 E5 07 1E 05 13 11 32 FF 80 00 00 02 10 01 10 02 04 12 00 28 09 06 00 08 19 09 00 FF 0F 02 12 00 00 02 04 12 00 28 09 06 00 08 19 09 00 FF 0F 01 12 00 00 02 04 12 00 01 09 06 00 00 2A 00 00 FF 0F 02 12 00 00 02 04 12 00 01 09 06 00 00 60 01 01 FF 0F 02 12 00 00 02 04 12 00 08 09 06 00 00 01 00 00 FF 0F 02 12 00 00 14 99 7E 7E A0 7D CE FF 03 13 D0 45 E0 40 00 02 00 00 6C 02 04 12 00 03 09 06 01 00 01 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 02 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 03 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 04 07 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 01 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 02 08 00 FF 0F 02 12 00 00 B3 98 7E 7E A0 8B CE FF 03 13 EE E1 E0 40 00 03 00 00 7A 02 04 12 00 03 09 06 01 01 05 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 06 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 07 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 01 08 08 00 FF 0F 02 12 00 00 02 04 12 00 03 09 06 01 00 0D 07 00 FF 0F 02 12 00 00 09 06 00 08 19 09 00 FF 09 10 4C 47 5A 31 30 33 30 36 35 35 39 33 33 35 31 32 09 07 31 39 33 35 3B 2A 7E 7E A0 57 CE FF 03 13 E9 69 E0 C0 00 04 00 00 46 39 31 32 09 0C 07 E5 07 1E 05 13 11 34 FF 80 00 81 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 00 00 00 06 00 0D 89 15 06 00 00 00 00 06 00 00 00 12 06 00 00 00 01 06 00 00 00 00 06 00 04 72 28 12 03 E8 86 C1 7E
Thanks and best regards,
raymar
Hi Raymar,
Hi Raymar,
Thanks for pointing this out. This is now fixed and the new version (107) is released.
BR,
Mikko
Hi Mikko
Hi Mikko
Thank you for handling this so quickly and your fast response. Now, the software runs with both smart meters. I really appreciate the great support you provide!
Best regards,
raymar