From your question I am not able to get if you have a default category view or a layered category view.
Update the same thing in <catalog_category_default> tag.
Also, update your code like below.

<block type="page/html_pager" name="product_list_toolbar_pager">
    <action method="setDefaultListPerPage"><limit>6</limit></action>
    <action method="setDefaultGridPerPage"><limit>6</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>6</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>12</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>24</limit></action>
<block>

You need to set these values inside product_list_toolbar_pager block. More reference

From your question I am not able to get if you have a default category view or a layered category view.
Update the same thing in <catalog_category_default> tag.
Also, update your code like below.

<block type="page/html_pager" name="product_list_toolbar_pager">
    <action method="setDefaultListPerPage"><limit>6</limit></action>
    <action method="setDefaultGridPerPage"><limit>6</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>6</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>12</limit></action>
    <action method="addPagerLimit"><mode>list</mode><limit>24</limit></action>
<block>

You need to set these values inside product_list_toolbar_pager block. More reference