With the wide expansion of voice services over the IP networks (VoIP), the volume of this delay sensitive traffic is steadily growing. The current packet schedulers for IP networks meet the delay constraint of VoIP traffic by simply assigning its packets the highest priority. This technique is acceptable as long as the amount of VoIP traffic is relatively very small compared to other non-voice traffic. With the notable expansion of VoIP applications, however, the current packet schedulers will significantly sacrifice the fairness deserved by the non-voice traffic. In this paper, we extend the conventional Deficit Round-Robin (DRR) scheduler by including a packet classifier, a Token Bucket and a resource reservation scheme and propose an integrated packet scheduler architecture for the growing VoIP traffic. We demonstrate through both theoretical analysis and extensive simulation that the new architecture makes it possible for us to significantly improve the fairness to non-voice traffic while still meeting the tight delay requirement of VoIP applications.