Re: RFC: Internally Generated "Records"

[Evan Huus <eapache () gmail com>]

On Mon, Aug 4, 2014 at 4:25 PM, Roland Knall <rknall () gmail com> wrote:

> Hello Evan
>
> Just a little side-note, could you explain what you mean by "records"?

I left it intentionally vague because of the various possible applications.
The one I tend to use as a "working example" when thinking about this is
protocols over TCP; you can have multiple application-layer frames inside a
single TCP packet, something like:

TCP Header
\
|| - application frame 1
|| - application frame 2
|| - application frame 3

Right now you can't filter on field combinations that must appear
"together" in one of those application frames: if fieldA appears in frame
1, and fieldB appears in frame 2, then that packet will match "fieldA &&
fieldB" even if they never appear "together" in the way a normal human
would intend. Being able to label each of those frames as a separate
"record" would solve this problem.


> With the openSAFETY dissector I voiced the issue some time ago, that
> openSAFETY in itself is a protocol, where it may end up being multiple
> nodes sending data in the same ethernet frame. Your solutions seem similar,
> although I still would face one issue, and that is visual representation.

Again, I specifically left that part vague :)

I am trying to solve the how-do-we-store-this-data problem; how we then
display that to the user is a separate, though related problem.


> On a field-bus network (Powerlink, SercosIII, Ethercat, openSAFETY over
> XXX, ...) there are devices called bus-controllers, which basically gateway
> data from backplane connected devices onto an ethernet-based bus. Therefore
> we would have ethernet frames, where we would see up to 500 nodes crammed
> into one single Ethernet frame (for the openSAFETY dissector it would be
> around 50). Simply stating where a new record starts is not enough, because
> there may be data in between (for managing the cramming of the nodes), the
> data may be multiplexed over multiple frames, and so on.

The current proposal would permit marking what are currently particular
subtrees as new records - anything *under* the subtree (not necessarily
*after* it) would be part of the new record, so you could quite reasonably
make a tree that looked like:

OpenSAFETY
\
|| - Node1 (new record)
|||| \- some fields under the Node1 record (since their parent is the Node1
subtree)
|| - some intermediary framing information, under the original record
(since its parent is the protocol root)
|| - Node2 (new record)
|||| \- some fields under the Node2 record (since their parent is the Node2
subtree)


> Also, beside the obvious data representation, the visual representation is
> lacking big-time. Take a look at any openSAFETY trace I put online, and
> you'll see what I mean. On a daily basis (at my work) I see Ethernet frames
> containing 30-50 openSAFETY nodes on smaller networks, which leads to a
> nearly impossible to decipher picture. I could provide you with such a
> sample trace, to further demonstrate the issue. My best answer to that
> problem (and I have been thinking on it for months now), would be some
> sub-entry representation in the list, similar to a directory structure.
>
> That said, I started a three weeks vacation today, mostly for working on a
> solution to my problem above, and as well as to getting the extcap
> interface finally out of gerrit. So if I can be of any help, feel free to
> ask, I would really appreciate a solution going forward towards 2.0.

The nice part about the proposal I have "so far" is that it's really only a
few lines to add the data - then somebody more motivated than I can write
code in the GUI and display filter to actually *use* the data. If there's
consensus that this method of storing the data makes sense then I'll add
it, and you're free to use it however you want.


> regards,
> Roland
>
>
> On Mon, Aug 4, 2014 at 9:56 PM, Evan Huus <eapache () gmail com> wrote:
>
> > One of the issues that's been popping up a lot recently is how to handle
> > "packets" that contain multiple "records". The reason both those words are
> > in quotes is because there's some broader context and applications:
> >
> >  - Putting each application-layer PDU into its own "record" regardless of
> > higher-level grouping or reassembly (by e.g. TCP) would theoretically
> > permit filtering for packets where only (fieldA==foo && fieldB==bar) occur
> > together in the same "record", and not just in the same on-the-wire packet.
> >
> >  - It opens up interesting opportunities for the summary list to be able
> > to, for example, hide everything above the application layer and only
> > display application-layer "records" (again, regardless of TCP-level
> > grouping or fragmentation).
> >
> >  - It is a necessary component of dissecting record-based file formats
> > with the current plan of reading the entire file as a single "packet".
> >
> > I've been thinking about this and trying to come up with a way to
> > gracefully (and backwards-compatibly) add this information to our existing
> > data-structures, and I'm currently thinking of just adding it as a flag to
> > the field_info struct (i.e. defining something like FI_STARTS_NEW_RECORD).
> > As far as I've been able to determine, these flag values are accessible
> > everywhere we need them to be (specifically: in the UI and in the
> > display-filter engine), and it makes creating new records
> > backwards-compatible and cheap (just some new macro to set the flag).
> >
> > My only concern right now is how difficult it will actually be to check
> > this value in the display filter logic - I don't know nearly enough about
> > the dfvm to know if checking for fields in the same "record" is easy or not
> > with the info stored this way.
> >
> > Thoughts? Suggestions? Better ideas?
> >
> >
> > ___________________________________________________________________________
> > Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
> > Archives:    http://www.wireshark.org/lists/wireshark-dev
> > Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
> >              mailto:wireshark-dev-request () wireshark org
> > ?subject=unsubscribe
>
>
> ___________________________________________________________________________
> Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
> Archives:    http://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
>              mailto:wireshark-dev-request () wireshark org
> ?subject=unsubscribe
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe