Keaton the oldest bangs on the drums and can play a few chords on my guitar and Callen the youngest also plays some on the drums both can hold a beat at the ages of 1 and 3!
Bounce then on your lap or whatever to music you like with the beat of what you are listening too...
Their brains will stop wiring for music right around age 5, but anything you can do to reinforce rhythm (or pitch, like listening to classical music or opera) will help them. Once kids are around 1st grade, you can build on what they have musically, but there isn't nearly as much 'wiring' you can continue to add...
If you're not a singer, at least rhythm is an easy one... have them clap with you a steady beat (if they can), and then double time it (also if they can), i.e. if you're clapping once a second (60 beats a minute) start clapping twice a second (120 beats a second) you'll be teaching them how to subdivide from quarter notes to eighth notes...