Wireshark mailing list archives

Re: hf_http_response_code in packet-http.c

From: "Sultan, Hassan via Wireshark-dev" <wireshark-dev () wireshark org>
Date: Thu, 13 Jul 2017 20:41:11 +0000

-----Original Message-----
From: Pascal Quantin [mailto:pascal.quantin () gmail com]
Sent: Thursday, July 13, 2017 1:16 PM
To: Developer support list for Wireshark <wireshark-dev () wireshark org>
Cc: Sultan, Hassan <sultah () amazon com>
Subject: Re: [Wireshark-dev] hf_http_response_code in packet-http.c



Le 13 juil. 2017 22:00, "Sultan, Hassan via Wireshark-dev" <wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > a écrit :




      > -----Original Message-----
      > From: Wireshark-dev [mailto:wireshark-dev-bounces () wireshark org
<mailto:wireshark-dev-bounces () wireshark org> ] On Behalf
      > Of Erik de Jong

      > Sent: Thursday, July 13, 2017 11:58 AM
      > To: Developer support list for Wireshark <wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> >
      > Subject: Re: [Wireshark-dev] hf_http_response_code in packet-http.c
      >
      >
      >

      > On Thu, Jul 13, 2017 at 8:47 PM, Sultan, Hassan via Wireshark-dev
<wireshark-
      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > > wrote:
      >
      >
      >
      >
      >       > -----Original Message-----
      >       > From: Pascal Quantin [mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com>
      > <mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com> > ]

      >       > Sent: Thursday, July 13, 2017 11:26 AM
      >       > To: Sultan, Hassan <sultah () amazon com
<mailto:sultah () amazon com>

      > <mailto:sultah () amazon com <mailto:sultah () amazon com> > >
      >       > Cc: Developer support list for Wireshark <wireshark-
      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > >

      >       > Subject: RE: [Wireshark-dev] hf_http_response_code in packet-
http.c
      >       >
      >       >
      >       >
      >       > Le 13 juil. 2017 19:54, "Sultan, Hassan" <sultah () amazon com
<mailto:sultah () amazon com>
      > <mailto:sultah () amazon com <mailto:sultah () amazon com> >

      >       > <mailto:sultah () amazon com <mailto:sultah () amazon com>
<mailto:sultah () amazon com <mailto:sultah () amazon com> > > > a

      > écrit :
      >       >
      >       >
      >       >
      >       >
      >       >       > -----Original Message-----
      >       >       > From: Pascal Quantin [mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com>
      > <mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com> >
      >       > <mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com>
      > <mailto:pascal.quantin () gmail com
<mailto:pascal.quantin () gmail com> > > ]
      >       >       > Sent: Thursday, July 13, 2017 10:39 AM
      >       >       > To: Developer support list for Wireshark <wireshark-

      >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >  <mailto:wireshark-
<mailto:wireshark->
      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > > >
      >       >

      >       >       > Cc: Sultan, Hassan <sultah () amazon com
<mailto:sultah () amazon com>
      > <mailto:sultah () amazon com <mailto:sultah () amazon com> >

      >       > <mailto:sultah () amazon com <mailto:sultah () amazon com>
<mailto:sultah () amazon com <mailto:sultah () amazon com> > > >

      >       >       > Subject: Re: [Wireshark-dev] hf_http_response_code in
packet-
      > http.c
      >       >       >
      >       >
      >       >       > Hi Hassan,
      >       >       >
      >       >       >
      >       >       > 2017-07-13 19:25 GMT+02:00 Sultan, Hassan via Wireshark-
dev
      >       > <wireshark-
      >       >
      >       >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >

      > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> > >
<mailto:wireshark- <mailto:wireshark->
      > <mailto:wireshark- <mailto:wireshark-> >
      >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >  <mailto:wireshark-
<mailto:wireshark->

      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > > > >:
      >       >
      >       >       >
      >       >       >
      >       >       >
      >       >       >
      >       >       >       > -----Original Message-----
      >       >       >       > From: Wireshark-dev [mailto:wireshark-dev-
<mailto:wireshark-dev->

      > <mailto:wireshark-dev- <mailto:wireshark-dev-> >
      >       > bounces () wireshark org <mailto:bounces () wireshark org>
<mailto:bounces () wireshark org <mailto:bounces () wireshark org> >
      > <mailto:wireshark-dev-bounces () wireshark org <mailto:wireshark-
dev-bounces () wireshark org>  <mailto:wireshark-dev- <mailto:wireshark-dev->
      > bounces () wireshark org <mailto:bounces () wireshark org> > >
      >       >       > <mailto:wireshark-dev-bounces () wireshark org
<mailto:wireshark-dev-bounces () wireshark org>
      > <mailto:wireshark-dev-bounces () wireshark org <mailto:wireshark-
dev-bounces () wireshark org> >  <mailto:wireshark- <mailto:wireshark->

      > <mailto:wireshark- <mailto:wireshark-> >
      >       > dev-bounces () wireshark org <mailto:dev-
bounces () wireshark org>  <mailto:dev-bounces () wireshark org <mailto:dev-
bounces () wireshark org> > >
      > > ] On Behalf
      >       >       >       > Of Erik de Jong
      >       >       >       > Sent: Wednesday, July 12, 2017 10:12 PM
      >       >       >       > To: Developer support list for Wireshark <wireshark-
      >       >
      >       >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >

      > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> > >
<mailto:wireshark- <mailto:wireshark->
      > <mailto:wireshark- <mailto:wireshark-> >
      >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >  <mailto:wireshark-
<mailto:wireshark->
      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > > > >
      >       >       >       > Subject: Re: [Wireshark-dev] hf_http_response_code in
      > packet-
      >       > http.c
      >       >       >       >
      >       >       >       >
      >       >       >       >
      >       >       >       > On Thu, Jul 13, 2017 at 1:12 AM, Sultan, Hassan via
      > Wireshark-
      >       > dev
      >       >       > <wireshark-
      >       >
      >       >       >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >
      > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> > >

      >       > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >

      > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> > > >
      > <mailto:wireshark- <mailto:wireshark->  <mailto:wireshark-
<mailto:wireshark-> >
      >       > <mailto:wireshark- <mailto:wireshark->  <mailto:wireshark-
<mailto:wireshark-> > >
      >
      >       >
      >       >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >
      > <mailto:dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> > >
<mailto:wireshark- <mailto:wireshark->
      > <mailto:wireshark- <mailto:wireshark-> >
      >       > dev () wireshark org <mailto:dev () wireshark org>
<mailto:dev () wireshark org <mailto:dev () wireshark org> >  <mailto:wireshark-
<mailto:wireshark->
      > dev () wireshark org <mailto:dev () wireshark org>  <mailto:wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org> > > > > > wrote:
      >       >       >       >
      >       >       >       >
      >       >       >       >       Hi,
      >       >       >       >
      >       >       >       >
      >       >       >       >
      >       >       >       >       I am starting to learn the Wireshark code base, and
one
      > thing
      >       >       > puzzles
      >       >       >       > me…
      >       >       >       >
      >       >       >       >
      >       >       >       >
      >       >       >       >       Why is hf_http_response_code defined as a
FT_UINT16
      > with
      >       >       > BASE_DEC
      >       >       >       > rather than an FT_STRING ?
      >       >       >       >
      >       >       >       >
      >       >       >       >
      >       >       >       >       It’s a text field… not an integer.
      >       >       >       >
      >       >       >       >
      >       >       >       > Presenting it as a number allows for filtering like:
      >       >       >       > http.response.code > 200
      >       >       >       >
      >       >       >       > Which would not be possible when presented as a
string.
      >       >       >
      >       >       >       Thanks for the info, but in that case would it not be more
      >       > appropriate to
      >       >       > have the normal field as an FT_STRING and add a generated
field
      > as
      >       > FT_UINT16
      >       >       > ? My understanding of generated fields is that this is their
      > purpose :
      >       > represent
      >       >       > data that doesn't exactly correspond to the packet data.
      >       >       >       We could still keep the field named as is today (hence
      > ensuring all
      >       >       > existing filters still work), but simply make it a generated
field,
      > and add
      >       > an
      >       >       > FT_STRING to represent the actual data as it is in the packet.
      >       >       >
      >       >       >       Thoughts ?
      >       >       >
      >       >       >
      >       >       >
      >       >       >
      >       >       > Does having the field as it is designed today generating you
any
      > issue?
      >       > For fields
      >       >       > having numerical values only, it makes much more sense to
have
      > the
      >       > digits
      >       >       > directly instead of the string and that's already what we are
doing
      > in
      >       > numerous
      >       >       > places. And personally I do not see any valid reason to
change
      > that.
      >       > Your
      >       >       > suggestion implies that many fields would start to be
duplicated
      >       > without any
      >       >       > added value.
      >       >
      >       >
      >       >       My reason is that anyone wanting to build automation based
on
      >       > Wireshark's parsing will not be able to rely on Wireshark's
reporting of
      > field type
      >       > & format to interpret the packet data.
      >       >       The automation would walk through parsed fields and be told
that
      > the
      >       > http response code is a 3-bytes UINT16 with value say 200, so
bytes 00
      > 00 c8,
      >       > which is wrong. Right now the way these fields are setup works
for
      > human
      >       > interpretation, by people looking at the rendered value on a
screen,
      > but it is far
      >       > from ideal for automation.
      >       >
      >       >
      >       > Thanks fact that HTTP is text based does not change the fact that
the
      > code is
      >       > supposed to be a number. So I'm not sure to get your point.
      >       > Moreover tons of people use tshark for automation and I do not
      > remember
      >       > someone complaining about the current output. So maybe I
simply did
      > not
      >       > understand your use case.
      >       >
      >       >
      >       >       One could argue it's the main purpose of Wireshark, but I
think it
      >       > needlessly prevent a whole class of use-cases from being
achieved
      > with
      >       > Wireshark's technology.
      >       >
      >       >
      >       >
      >       > Could you elaborate? What use case do you have in mind
exactly?
      >
      >
      >       Imagine automation that attempts to perform modification on
packets
      > using the parsed information provided by Wireshark, or somehow
goes and
      > directly reads fields in the packet data based on what Wireshark's
parsed
      > information says (type/offset/length/...). Wireshark reporting the field
as
      > FT_UINT16 instead of say a UTF-8 FT_STRING throws off the
automation as it
      > would either attempt to write an FT_UINT16 at the offset of the field,
writing
      > the wrong format, or be confused by seeing a 3 bytes FT_UINT16 and
give up.
      >
      >
      >
      >
      > But in that case it's more like you're doing a part of the parsing in
another
      > application, why not implement a full parser in that application? The
whole point
      > of an analyzer is to make sense of the data passed into it for the user.
And what
      > about protocols that could be send with different encodings (BER,
XER, etc)
      > there a field type should still be the same - eg UINT16 - but the
representation
      > on the wire is totally different.


      The beauty of this is specifically that the other app does not need to do
any parsing at all. It fully relies on Wireshark's parsing engine to get the tree of
fields, the offset/length/type/name/encoding/... of each field and instead of
having to write custom code for each protocol (you know, a parser), the app
simply has to implement generic handling for FT_STRING, FT_UINT16, etc... and
the various encodings. So for your BER encoded integer, having an FT_UINT16
with an ENC_BER encoding flag for example and a size of 4 (type+length+2 bytes
for most short ints) would convey all the info needed to handle the field.



That would require having an encoder able to understand Wireshark output
syntax and the structure of the protocol you are using to perform the encoding
(text based, TLV, CSN1, ASN1, etc): all the encoding rules related to the protocol
you are playing with. Having a decoder / parser is far from giving you an
encoder.


Absolutely, but this is a lot, lot, less work than writing parsers for all the protocols. That's where Wireshark 
shines, it provides 70-80% of the work (all the parsing), and only the handling of encoding / shifting fields (in case 
user decides to write a field to be larger/smaller than originally in the packet) needs to be done.

As Eric explained, having the HTTP response code as a number simply gives you
much more advantages (filtering, comparisons, inequalities...) than having it in
text. So this is clearly not a bug as it was done on purpose.


I agree it's not a bug. For the existing purpose of Wireshark it does the job. It just makes some other use-cases a bit 
more difficult.

      Will such a thing work for absolutely every single protocol out there ?
Maybe not, but it works for the vast majority of them.


I have some doubts regarding this statement, for the reason I gave above.
But as I'm eager to learn, if you can come up with a PoC I would be very
interested to look at it :)

 
It might be possible to show something but it will take a bit of time and is a maybe. I'm still in the learning stages 
of how Wireshark works and there's quite a bit of work that will be needed before I can confirm if it's doable with 
Wireshark or what changes are needed in Wireshark to make it work (the example I just gave being one).

      I did a very similar thing on top of Network Monitor in a previous life at
Microsoft (I had the code available being internal to the company) and it was a
huge success, helping many people in various ways.



So what kind of encoding did it support?


Gzip compression, asn.1 BER & DER, utf-16/8/ucs2/ascii, little/big endian, base64, custom encodings for netbios 
strings, and more that I forget, it also had inference for finer grained datatypes (figure out that this string field 
is actually a url, domain name, numerical text, email, xml, json, ... instead of text and it did so in a recursive 
manner so that when finding xml for example, attributes in xml could be inferred as url, numerical text, domain name, 
etc...). It could also infer which fields were length/count fields for other fields (so it could update them 
automatically upon changes) or, in certain circumstances, checksum fields. Whenever users on a protocol encountered an 
encoding we didn't support, we'd add the encoder for it. Much simpler and quicker task than adding a protocol parser.



      >       How about having an #ifdef for such things so that Wireshark's
use-case
      > today does not have to suffer the addition of generated fields, and
such
      > automation cases could still get full fidelity parsing by building
Wireshark's
      > engine with the specific #define ? Today they have to build directly
against
      > Wireshark's code anyway so that shouldn't be a problem.
      >
      >
      >
__________________________________________________________
      > _________________
      >       Sent via:    Wireshark-dev mailing list <wireshark-
dev () wireshark org <mailto:wireshark-dev () wireshark org>

      > <mailto:wireshark-dev () wireshark org <mailto:wireshark-
dev () wireshark org> > >
      >       Archives:    https://www.wireshark.org/lists/wireshark-dev
<https://www.wireshark.org/lists/wireshark-dev>
      > <https://www.wireshark.org/lists/wireshark-dev
<https://www.wireshark.org/lists/wireshark-dev> >
      >       Unsubscribe:
https://www.wireshark.org/mailman/options/wireshark-
<https://www.wireshark.org/mailman/options/wireshark->
      > dev <https://www.wireshark.org/mailman/options/wireshark-dev
<https://www.wireshark.org/mailman/options/wireshark-dev> >
      >                    mailto:wireshark-dev-request () wireshark org
<mailto:wireshark-dev-request () wireshark org>  <mailto:wireshark-
<mailto:wireshark->
      > dev-request () wireshark org <mailto:dev-request () wireshark org> >
?subject=unsubscribe
      >


      __________________________________________________________
_________________
      Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org
<mailto:wireshark-dev () wireshark org> >
      Archives:    https://www.wireshark.org/lists/wireshark-dev
<https://www.wireshark.org/lists/wireshark-dev>
      Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-
dev <https://www.wireshark.org/mailman/options/wireshark-dev>
                   mailto:wireshark-dev-request () wireshark org <mailto:wireshark-
dev-request () wireshark org> ?subject=unsubscribe


___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    https://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://www.wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe

Current thread:

hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 12)
- Re: hf_http_response_code in packet-http.c Erik de Jong (Jul 12)
  - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 13)
    - Re: hf_http_response_code in packet-http.c Pascal Quantin (Jul 13)
    - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 13)
    - Re: hf_http_response_code in packet-http.c Pascal Quantin (Jul 13)
    - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 13)
    - Re: hf_http_response_code in packet-http.c Erik de Jong (Jul 13)
    - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 13)
    - Re: hf_http_response_code in packet-http.c Pascal Quantin (Jul 13)
    - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 13)
    - Re: hf_http_response_code in packet-http.c Jaap Keuter (Jul 15)
    - Re: hf_http_response_code in packet-http.c Sultan, Hassan via Wireshark-dev (Jul 17)
    - Re: hf_http_response_code in packet-http.c Anders Broman (Jul 17)
    - Re: hf_http_response_code in packet-http.c Guy Harris (Jul 17)