Laravel, Dusk and Valet

By default I set my phpunit.xml to use test as my database:

    <php>
        <env name="APP_ENV" value="testing"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="SESSION_DRIVER" value="array"/>
        <env name="MAIL_DRIVER" value="log"/>
        <env name="QUEUE_DRIVER" value="sync"/>
        <env name="DB_DATABASE" value="test"/> <-- this line here 
    </php>

But with Valet and Dusk this led to the browser using my default database and not test but the PHPUnit test itself was using test so the user I created and logged in as was not available in the browser.

I tried to set .env.dusk.local and I tried phpuni.dusk.xml none of it worked so I had to remove <env name="DB_DATABASE" value="test"/>

This is a bummer and I am sure I am missing something. What this means and my tests reset my database for my local app which ideally it would not since it is really nice to keep state.