I have a D5500 and that's normal download time for a 10/100mb/s network speccially using NEF(raw) format. Using Native format is a bit faster but maybe inconvenient
My Canon 550D also did take about the same amount of time.
Now that I have a 1Gbit network it takes about 5 seconds to download my images from my atik to my room.