Re: pte fix

[pedward () WEBCOM COM (pedward () WEBCOM COM)]

Wayne,

 (I'm ccing this response to bugtraq).

Setrlimit will not work to prevent this DoS attack.  This attack works
because of 2 things:

Each process has a fixed amount of data that doesn't get swapped to disk.
The process task structure and the page table lists don't get swapped out.
Linux maintains a 3 tier page table structure for memory allocation to
each process.  This table has a single 4 byte integer for each entry. Page
tables use a bitfield encoding mechanism of a 32 bit (or 64 bit) pointer.

The problem is that when the mmap is done, the page table chain is expanded.
When the munmap is done, the page_table isn't collapsed.  A usage count
could be maintained for the PTEs, since anything that touches them is
possibly an "expensive" task.

The page tables are used during the swapping process to mark which pages
have been swapped out and such.  A process maintains (on x86) 1 page
table for every 4MB of RAM, then a lower level page table is used to
represent every page in a 4MB chunk.  I suppose that it's possible to
swap out the page table in deep swap mode, but that has a lot of nasty
things.

So, you could limit the number of processes a person could use, limiting
the possible effect of such a DoS.  The page tables consume 768*sizeof(pte_t)
bytes, assuming the program that's the subject of this discussion.

I'm doing some more investigation on this. The source indicates that it should
be freeing the PTEs when the munmap is called.

--Perry

> you mentioned something about using setrlimit and such to fix this
> problem.. I have been screwing around with it for a while now and cannot
> get it to change anything.  Could you please explain how you would do it?
> -- i tried setting the rlim_cur and max -- don't know what i am doing
> wrong.
>
> Wayne
>
>
> +-=---==- -- -=- ---=------=-=- - -=---------==- -===-- -- - --==--=+
> | NT vs Linux, why Linux:                                           |
> |   It doesn't matter how big, hot, or easy your *thing* is if it   |
> |   doesn't stay up, perform, or handle it's load.                  |
> +---==-- - --=-- -=---- -=-------=-==-=====-- --=-- -- - --=- -===--+


--
Perry Harrington        System Software Engineer    zelur xuniL  ()
http://www.webcom.com  perry.harrington () webcom com  Think Blue.  /\